Jul 02 12:37:08.714815 ip-10-0-143-149 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 02 12:37:08.714827 ip-10-0-143-149 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 02 12:37:08.714837 ip-10-0-143-149 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 02 12:37:08.715238 ip-10-0-143-149 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 02 12:37:18.866144 ip-10-0-143-149 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 02 12:37:18.866161 ip-10-0-143-149 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 35e059bad25a4597acd87611a098aea9 --
Jul 02 12:39:10.535838 ip-10-0-143-149 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.923868    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927925    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927938    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927942    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927944    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927948    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927951    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927954    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:39:11.003927 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927957    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:39:10.994357 ip-10-0-143-149 systemd[1]: Started Kubernetes Kubelet.
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927960    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927962    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927965    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927968    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927971    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927973    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927977    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927979    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927982    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927985    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927988    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.927992    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928001    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928005    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928008    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928011    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928014    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928017    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:39:11.006204 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928020    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928022    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928026    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928029    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928032    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928035    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928038    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928040    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928043    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928046    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928048    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928051    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928054    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928056    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928059    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928062    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928064    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928067    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928070    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928072    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:39:11.012114 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928075    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928077    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928079    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928082    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928084    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928087    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928091    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928094    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928096    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928099    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928102    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928104    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928107    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928110    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928113    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928116    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928119    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928123    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928127    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928130    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:39:11.015230 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928132    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928135    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928138    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928141    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928143    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928146    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928148    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928151    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928153    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928156    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928159    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928162    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928164    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928167    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928170    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928172    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928175    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928178    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928180    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928183    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:39:11.017702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928186    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928188    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928191    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928193    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928197    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928199    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928204    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928207    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928210    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928214    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928216    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928219    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.928222    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929748    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929756    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929760    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929763    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929767    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929770    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:39:11.020494 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929772    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929775    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929778    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929780    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929783    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929786    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929789    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929791    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929794    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929797    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929799    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929802    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929804    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929807    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929810    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929813    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929817    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929819    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929822    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929825    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:39:11.023197 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929827    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929830    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929833    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929836    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929838    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929841    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929844    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929847    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929851    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929854    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929859    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929863    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929866    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929869    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929872    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929874    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929877    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929879    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929882    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:39:11.025699 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929885    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929887    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929890    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929893    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929895    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929898    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929902    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929905    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929908    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929911    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929914    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929916    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929919    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929922    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929924    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929927    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929930    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929933    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929937    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:39:11.027585 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929941    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929945    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929948    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929951    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929954    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929957    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929959    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929962    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929964    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929967    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929970    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929972    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929975    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929977    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929980    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929983    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929985    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929988    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929990    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929996    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:39:11.029990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.929998    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930001    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930004    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930006    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930009    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930012    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930014    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930017    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930020    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930022    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930025    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930027    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930030    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930032    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930035    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930420    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930427    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930430    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930433    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:39:11.031280 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930436    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930439    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930441    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930452    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930455    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930458    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930461    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930464    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930466    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930469    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930472    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930474    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930478    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930481    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930484    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930488    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930491    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930495    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930498    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930501    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:39:11.181652 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930503    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930506    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930509    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930511    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930514    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930516    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930519    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930521    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930524    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930526    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930529    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930532    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930552    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930557    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930561    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930565    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930567    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930570    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930572    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930575    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:39:11.199971 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930577    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930580    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930583    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930585    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930589    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930592    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930594    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930597    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930600    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930605    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930607    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930610    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930612    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930615    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930618    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930620    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930623    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930625    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930628    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930630    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:39:11.202024 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930633    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930636    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930638    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930640    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930643    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930646    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930648    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930651    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930653    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930656    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930658    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930660    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930663    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930665    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930668    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930671    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930674    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930677    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930679    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930682    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:39:11.203042 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930684    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930689    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930693    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930697    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930700    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930703    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930705    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930708    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930711    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930713    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930715    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930718    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930720    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930723    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.930726    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930791    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930802    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930808    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930813    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930817    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930821    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 12:39:11.204840 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930825    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930829    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930832    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930835    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930838    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930842    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930845    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930849    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930852    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930855    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930858    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930861    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930866    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930870    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930873    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930876    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930880    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930883    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930886    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930890    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930893    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930896    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930899    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930902    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 12:39:11.206384 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930905    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930908    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930912    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930915    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930917    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930920    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930923    2564 flags.go:64] FLAG: --enable-server="true"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930926    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930932    2564 flags.go:64] FLAG: --event-burst="100"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930935    2564 flags.go:64] FLAG: --event-qps="50"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930937    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930940    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930943    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930947    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930951    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930955    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930958    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930961    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930963    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930967    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930971    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930974    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930977    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930980    2564 flags.go:64] FLAG: --feature-gates=""
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930984    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 12:39:11.207737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930987    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930990    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930994    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930997    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.930999    2564 flags.go:64] FLAG: --help="false"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931002    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931005    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931008    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931011    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931014    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931018    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931021    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931023    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931026    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931029    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931032    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931035    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931038    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931041    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931043    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931046    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931049    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931053    2564 flags.go:64] FLAG: --lock-file=""
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931056    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 12:39:11.208854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931059    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931062    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931067    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931071    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931074    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931077    2564 flags.go:64] FLAG: --logging-format="text"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931080    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931084    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931087    2564 flags.go:64] FLAG: --manifest-url=""
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931089    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931094    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931097    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931100    2564 flags.go:64] FLAG: --max-pods="110"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931103    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931106    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931109    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931112    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931115    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931118    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931121    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931127    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931130    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931133    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 12:39:11.209852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931136    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931139    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931144    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931148    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931151    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931154    2564 flags.go:64] FLAG: --port="10250"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931157    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931160    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00f9213da97909299"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931165    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931168    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931171    2564 flags.go:64] FLAG: --register-node="true"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931173    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931178    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931181    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931184    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931187    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931191    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931194    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931197    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931200    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931203    2564 flags.go:64] FLAG: --runonce="false"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931206    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931209    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931212    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931214    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 12:39:11.210941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931217    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931220    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931223    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931226    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931229    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931233    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931235    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931238    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931241    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931244    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931247    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931252    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931254    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931257    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931261    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931265    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931268    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931271    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931274    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931278    2564 flags.go:64] FLAG: --v="2"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931282    2564 flags.go:64] FLAG: --version="false"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931286    2564 flags.go:64] FLAG: --vmodule=""
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931289    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.931292    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931402    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:39:11.211720 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931405    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931408    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931410    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931413    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931416    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931418    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931421    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931423    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931426    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931429    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931431    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931434    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931436    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931438    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931441    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931444    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931446    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931449    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931451    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931454    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:39:11.401979 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931456    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931459    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931463    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931465    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931468    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931470    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931475    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931480    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931483    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931486    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931489    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931492    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931494    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931497    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931500    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931502    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931505    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931507    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931510    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931512    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:39:11.402864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931515    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931519    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931522    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931525    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931528    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931531    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931551    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931555    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931558    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931560    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931563    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931566    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931569    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931571    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931575    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931577    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931580    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931584    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931588    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:39:11.403414 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931591    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931594    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931597    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931599    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931602    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931605    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931608    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931611    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931613    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931616    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931618    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931621    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931624    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931627    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931629    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931632    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931634    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931637    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931639    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931642    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:39:11.403990 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931644    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931646    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931649    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931652    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931654    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931657    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931660    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931663    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931666    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931669    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931671    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931676    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931679    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931681    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931684    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931686    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931689    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931691    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:39:11.404560 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.931694    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.932253    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.941753    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.941768    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.942699    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.942712    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942774    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942779    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942783    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942786    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942789    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942792    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942794    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942797    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942800    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:39:11.405115 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942803    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942806    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942808    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942811    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942814    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942818    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942822    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942825    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942828    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942830    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942833    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942835    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942838    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942841    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942845    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942848    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942850    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942853    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942857    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942860    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:39:11.405556 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942863    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942865    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942868    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942870    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942873    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942875    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942878    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942881    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942883    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942886    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942889    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942892    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942894    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942897    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942899    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942902    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942904    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942907    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942910    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942913    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:39:11.406137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942915    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942918    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942920    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942923    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942925    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942928    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942931    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942934    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942936    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942939    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942941    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942944    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942947    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942949    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942952    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942954    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942957    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942960    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942963    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942966    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:39:11.406694 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942968    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942971    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942975    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942978    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942981    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942984    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942987    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942989    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942992    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.942994    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943000    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943003    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943006    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943008    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943011    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943013    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943016    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943018    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943021    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:39:11.407257 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943024    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943027    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943029    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943032    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943034    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943037    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943040    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943044    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943047    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943050    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943054    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.943059    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943175    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943181    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943184    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:39:11.407795 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943187    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943190    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943193    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943196    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943199    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943202    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943205    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943211    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943217    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943221    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943225    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943229    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943232    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943235    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943238    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943240    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943243    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943246    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943248    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:39:11.408264 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943251    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943254    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943256    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943259    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943262    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943264    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943267    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943269    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943272    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943275    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943277    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943280    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943282    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943285    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943288    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943290    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943293    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943295    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943298    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:39:11.408788 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943300    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943305    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943310    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943313    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943317    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943320    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943323    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943326    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943328    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943331    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943333    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943336    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943338    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943341    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943344    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943347    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943350    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943354    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943357    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:39:11.409316 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943359    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943362    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943365    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943368    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943370    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943373    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943376    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943378    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943381    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943383    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943386    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943388    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943391    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943393    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943397    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943401    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943404    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943406    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943409    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943412    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:39:11.409864 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943414    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943417    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943419    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943422    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943425    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943427    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943430    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943432    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943435    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943437    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943440    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943443    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943445    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943447    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943450    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943452    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943455    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943458    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:39:11.410423 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:10.943460    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:39:11.410961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.943465    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 12:39:11.410961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.943600    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 12:39:11.410961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.946443    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 12:39:11.410961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.947634    2564 server.go:1019] "Starting client certificate rotation"
Jul 02 12:39:11.410961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.947726    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 12:39:11.410961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.947763    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 12:39:11.410961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.949968    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 12:39:11.410961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.951968    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 12:39:11.410961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.963349    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 02 12:39:11.410961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.965384    2564 log.go:25] "Validated CRI v1 image API"
Jul 02 12:39:11.410961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.966233    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 12:39:11.410961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.970496    2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 dd831463-9b0c-4c3e-99d2-93772d272af4:/dev/nvme0n1p3 ec96192a-0f88-4290-a2c1-4614d391f4ee:/dev/nvme0n1p4]
Jul 02 12:39:11.411331 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.970518    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 12:39:11.411404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.976694    2564 manager.go:216] Machine: {Timestamp:2026-07-02 12:39:10.97601709 +0000 UTC m=+0.302250010 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090389 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec281d1b7893d57e192e088d5f326090 SystemUUID:ec281d1b-7893-d57e-192e-088d5f326090 BootID:35e059ba-d25a-4597-acd8-7611a098aea9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:67:68:71:bd:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:67:68:71:bd:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:68:d3:b5:14:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 12:39:11.411404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.976831    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 12:39:11.411404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.976903    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 12:39:11.411404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.978106    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 12:39:11.411404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.978131    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-149.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 12:39:11.411404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.978754    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 12:39:11.411404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.978764    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 12:39:11.411404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.978771    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.979265    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.979381    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.980003    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.985066    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.985208    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.985241    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.985255    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.985267    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.985280    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.985291    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.985652    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.985667    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.987458    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.988879    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.988906    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.989481    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.989510    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.989520    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.989527    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.989553    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.989560    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.989568    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.989573    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.989579    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.989584    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.989592    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.989597    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.990130    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.990137    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.993486    2564 server.go:1268] "Started kubelet"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.993584    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.993601    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.993676    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.994920    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:10.996295    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:10.999881    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.000708    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.000788    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.001597    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.001607    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.003764    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.004058    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 12:39:11.411784 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.004066    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.004157    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.004183    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-149.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.004356    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.003369    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.004767    2564 factory.go:55] Registering systemd factory
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.004783    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.004141    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-149.ec2.internal.18be795ee6ec7fea  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-149.ec2.internal,UID:ip-10-0-143-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-149.ec2.internal,},FirstTimestamp:2026-07-02 12:39:10.993440746 +0000 UTC m=+0.319673704,LastTimestamp:2026-07-02 12:39:10.993440746 +0000 UTC m=+0.319673704,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-149.ec2.internal,}"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.005045    2564 factory.go:153] Registering CRI-O factory
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.005060    2564 factory.go:223] Registration of the crio container factory successfully
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.005632    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.005666    2564 factory.go:103] Registering Raw factory
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.005681    2564 manager.go:1196] Started watching for new ooms in manager
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.006037    2564 manager.go:349] Starting recovery of all containers
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.011393    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.011512    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.019832    2564 manager.go:354] Recovery completed
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.025501    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.027387    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.027414    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.027426    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.027996    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.028008    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.028026    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.028056    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.029197    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.029217    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:39:11.413173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.029228    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:39:11.413961 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.029809    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-149.ec2.internal.18be795ee8f2ad72  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-149.ec2.internal,UID:ip-10-0-143-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-149.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-149.ec2.internal,},FirstTimestamp:2026-07-02 12:39:11.02740005 +0000 UTC m=+0.353632968,LastTimestamp:2026-07-02 12:39:11.02740005 +0000 UTC m=+0.353632968,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-149.ec2.internal,}"
Jul 02 12:39:11.413961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.030212    2564 policy_none.go:49] "None policy: Start"
Jul 02 12:39:11.413961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.030226    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 12:39:11.413961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.030235    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 12:39:11.413961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.032943    2564 policy_none.go:47] "Start"
Jul 02 12:39:11.413961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.036528    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-42wh7"
Jul 02 12:39:11.413961 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.037666    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-149.ec2.internal.18be795ee8f2f8b7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-149.ec2.internal,UID:ip-10-0-143-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-149.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-149.ec2.internal,},FirstTimestamp:2026-07-02 12:39:11.027419319 +0000 UTC m=+0.353652237,LastTimestamp:2026-07-02 12:39:11.027419319 +0000 UTC m=+0.353652237,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-149.ec2.internal,}"
Jul 02 12:39:11.413961 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.039924    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-149.ec2.internal.18be795ee8f3234a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-149.ec2.internal,UID:ip-10-0-143-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-149.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-149.ec2.internal,},FirstTimestamp:2026-07-02 12:39:11.027430218 +0000 UTC m=+0.353663136,LastTimestamp:2026-07-02 12:39:11.027430218 +0000 UTC m=+0.353663136,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-149.ec2.internal,}"
Jul 02 12:39:11.413961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.045151    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-42wh7"
Jul 02 12:39:11.413961 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.046775    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-149.ec2.internal.18be795ee8f2ad72\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-149.ec2.internal.18be795ee8f2ad72  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-149.ec2.internal,UID:ip-10-0-143-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-149.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-149.ec2.internal,},FirstTimestamp:2026-07-02 12:39:11.02740005 +0000 UTC m=+0.353632968,LastTimestamp:2026-07-02 12:39:11.029207977 +0000 UTC m=+0.355440898,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-149.ec2.internal,}"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.051109    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-149.ec2.internal.18be795ee8f2f8b7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-149.ec2.internal.18be795ee8f2f8b7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-149.ec2.internal,UID:ip-10-0-143-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-149.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-149.ec2.internal,},FirstTimestamp:2026-07-02 12:39:11.027419319 +0000 UTC m=+0.353652237,LastTimestamp:2026-07-02 12:39:11.029221951 +0000 UTC m=+0.355454870,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-149.ec2.internal,}"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.071702    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.072218    2564 manager.go:337] "Starting Device Plugin manager"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.072243    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.072251    2564 server.go:87] "Starting device plugin registration server"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.072484    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.072495    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.072711    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.072803    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.072811    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.073783    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.073813    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.073831    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.073867    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.073866    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.073916    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.075748    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.173223    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.174039    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"]
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.174106    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.174178    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.174207    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.174221    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.174247    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.175484    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.175512    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.175524    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.177800    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.177894    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.177925    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:39:11.571489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.178456    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.178483    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.178492    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.178461    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.178562    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.178575    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.182308    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.182340    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.183288    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.183316    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.183331    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.183801    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.183817    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-149.ec2.internal\": node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.217424    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-149.ec2.internal\" not found" node="ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.221895    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-149.ec2.internal\" not found" node="ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.229805    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.311746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3966a64f6cf985e92da6b4f85445732-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal\" (UID: \"f3966a64f6cf985e92da6b4f85445732\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.311778    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a9ee241c13dace2d3c74948d3ea9c6f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-149.ec2.internal\" (UID: \"6a9ee241c13dace2d3c74948d3ea9c6f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.311797    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3966a64f6cf985e92da6b4f85445732-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal\" (UID: \"f3966a64f6cf985e92da6b4f85445732\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.330462    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.411923    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3966a64f6cf985e92da6b4f85445732-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal\" (UID: \"f3966a64f6cf985e92da6b4f85445732\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.411948    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3966a64f6cf985e92da6b4f85445732-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal\" (UID: \"f3966a64f6cf985e92da6b4f85445732\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.411969    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a9ee241c13dace2d3c74948d3ea9c6f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-149.ec2.internal\" (UID: \"6a9ee241c13dace2d3c74948d3ea9c6f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573862 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.411997    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a9ee241c13dace2d3c74948d3ea9c6f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-149.ec2.internal\" (UID: \"6a9ee241c13dace2d3c74948d3ea9c6f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573862 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.412033    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3966a64f6cf985e92da6b4f85445732-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal\" (UID: \"f3966a64f6cf985e92da6b4f85445732\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573862 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.412033    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3966a64f6cf985e92da6b4f85445732-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal\" (UID: \"f3966a64f6cf985e92da6b4f85445732\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573862 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.431029    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:11.573862 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.521925    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573862 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:11.526118    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:11.573862 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.531704    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:11.749257 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.632297    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:11.749257 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.732881    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:11.906357 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.833563    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:13.433146 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:11.934081    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:13.433146 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.289978    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 12:39:13.433146 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.290124    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 12:39:13.433146 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:13.377335    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:13.433146 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.392483    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 12:34:12 +0000 UTC" deadline="2028-02-22 13:47:22.567734961 +0000 UTC"
Jul 02 12:39:13.433146 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.392505    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14401h8m9.175233309s"
Jul 02 12:39:13.433146 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.406615    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:39:13.433146 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.408008    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:39:13.433146 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.408044    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:39:13.433146 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.408056    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:39:13.598578 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:13.478515    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:13.598578 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.524966    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:39:13.598578 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:13.571805    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9ee241c13dace2d3c74948d3ea9c6f.slice/crio-628cdc613fd01db0dce57e8036fac16a6619e0aab064a4c9059c8a11cb14c928 WatchSource:0}: Error finding container 628cdc613fd01db0dce57e8036fac16a6619e0aab064a4c9059c8a11cb14c928: Status 404 returned error can't find the container with id 628cdc613fd01db0dce57e8036fac16a6619e0aab064a4c9059c8a11cb14c928
Jul 02 12:39:13.598578 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:13.573194    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3966a64f6cf985e92da6b4f85445732.slice/crio-ad28b91a4f72461dbac9fee5e6ceadcd83b28cc98fdb4d6c9ac8a91b1182b89a WatchSource:0}: Error finding container ad28b91a4f72461dbac9fee5e6ceadcd83b28cc98fdb4d6c9ac8a91b1182b89a: Status 404 returned error can't find the container with id ad28b91a4f72461dbac9fee5e6ceadcd83b28cc98fdb4d6c9ac8a91b1182b89a
Jul 02 12:39:13.598578 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.576873    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 12:39:13.598578 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:13.578768    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:13.754289 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:13.679259    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Jul 02 12:39:13.754289 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.711392    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:39:13.754289 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.744429    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:13.881420 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.759859    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:39:13.881420 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.770717    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 12:39:13.881420 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.771748    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"
Jul 02 12:39:13.881420 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:13.793014    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 12:39:14.291331 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.252060    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:39:14.329166 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.329137    2564 apiserver.go:52] "Watching apiserver"
Jul 02 12:39:14.338197 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.338174    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 12:39:14.338724 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.338698    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-pgvs9","openshift-multus/multus-additional-cni-plugins-j2cwd","openshift-network-diagnostics/network-check-target-r975p","openshift-ovn-kubernetes/ovnkube-node-qmx4p","kube-system/global-pull-secret-syncer-bx9gl","kube-system/konnectivity-agent-nmzpg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57","openshift-dns/node-resolver-lczx4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal","openshift-multus/multus-sgqpb","openshift-multus/network-metrics-daemon-v7gwd","openshift-network-operator/iptables-alerter-2bmr8","kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xhsqd"]
Jul 02 12:39:14.342126 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.341491    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.343821 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.343801    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:14.343901 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.343871    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:14.344711 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.344630    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 12:39:14.344829 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.344808    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 12:39:14.345491 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.345182    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4nfcj\""
Jul 02 12:39:14.345491 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.345177    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 12:39:14.345491 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.345181    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 12:39:14.345921 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.345813    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:14.345921 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.345881    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:14.348145 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.348127    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.350546 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.350524    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:14.350633 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.350600    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:14.350905 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.350750    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m5c52\""
Jul 02 12:39:14.351319 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.351228    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 12:39:14.351319 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.351253    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 12:39:14.352308 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.352277    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 12:39:14.352424 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.352400    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 12:39:14.352516 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.352499    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 12:39:14.352738 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.352317    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 12:39:14.353745 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.353459    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nmzpg"
Jul 02 12:39:14.356177 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.356159    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 12:39:14.356548 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.356530    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.357179 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.356780    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 12:39:14.357179 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.357027    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j6fcw\""
Jul 02 12:39:14.359401 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.359382    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lczx4"
Jul 02 12:39:14.360865 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.360847    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.363352 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363182    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c34fc4a6-58be-48e7-83db-1a0d5619031e-ovnkube-config\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.363352 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363228    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363576    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-registration-dir\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363620    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-device-dir\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363656    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-run-systemd\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363691    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-run-ovn-kubernetes\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363722    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c34fc4a6-58be-48e7-83db-1a0d5619031e-ovnkube-script-lib\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363756    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2zck\" (UniqueName: \"kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck\") pod \"network-check-target-r975p\" (UID: \"2697b9b4-7477-4e35-a9f3-298ed7a423ed\") " pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-systemd-units\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363809    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c34fc4a6-58be-48e7-83db-1a0d5619031e-env-overrides\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363851    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w958t\" (UniqueName: \"kubernetes.io/projected/c34fc4a6-58be-48e7-83db-1a0d5619031e-kube-api-access-w958t\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363882    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-socket-dir\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363911    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zwbc\" (UniqueName: \"kubernetes.io/projected/dc387565-3fa1-4271-a12e-dda0c699e3e4-kube-api-access-6zwbc\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363958    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-kubelet\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.363981    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-slash\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-run-openvswitch\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364048    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-run-ovn\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364074    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-node-log\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364110    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-log-socket\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364165    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-etc-selinux\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364216    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xkrd\" (UniqueName: \"kubernetes.io/projected/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-kube-api-access-7xkrd\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364226    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364243    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364272    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-var-lib-openvswitch\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c34fc4a6-58be-48e7-83db-1a0d5619031e-ovn-node-metrics-cert\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364458    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-cni-bin\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364488    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-sys-fs\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364518    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-run-netns\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364558    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 12:39:14.366681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364573    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-etc-openvswitch\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364596    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xrkkc\""
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364608    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-cni-netd\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364643    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.364718    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hh48r\""
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.365217    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.365393    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.365493    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.366005    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.366065    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.366156    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.366447    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-blgrb\""
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.367127    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 12:39:14.367321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.367309    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 12:39:14.370012 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.369807    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.372411 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.372338    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pgvs9"
Jul 02 12:39:14.373837 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.373377    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 12:39:14.374766 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.374512    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2bmr8"
Jul 02 12:39:14.375097 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.375078    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n5pnt\""
Jul 02 12:39:14.377673 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.377518    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 12:39:14.381933 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.380720    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8z6gk\""
Jul 02 12:39:14.381933 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.381074    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 12:39:14.381933 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.381352    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6sfvw\""
Jul 02 12:39:14.381933 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.381472    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 12:39:14.381933 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.381562    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 12:39:14.381933 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.381767    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 12:39:14.382665 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.382504    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 12:39:14.424053 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.424002    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal" event={"ID":"f3966a64f6cf985e92da6b4f85445732","Type":"ContainerStarted","Data":"ad28b91a4f72461dbac9fee5e6ceadcd83b28cc98fdb4d6c9ac8a91b1182b89a"}
Jul 02 12:39:14.426355 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.426287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal" event={"ID":"6a9ee241c13dace2d3c74948d3ea9c6f","Type":"ContainerStarted","Data":"628cdc613fd01db0dce57e8036fac16a6619e0aab064a4c9059c8a11cb14c928"}
Jul 02 12:39:14.427980 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.427614    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k44gk"
Jul 02 12:39:14.436931 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.436908    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k44gk"
Jul 02 12:39:14.448506 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.448483    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465212    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zck\" (UniqueName: \"kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck\") pod \"network-check-target-r975p\" (UID: \"2697b9b4-7477-4e35-a9f3-298ed7a423ed\") " pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-systemd-units\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-var-lib-cni-multus\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465325    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-os-release\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465352    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zwbc\" (UniqueName: \"kubernetes.io/projected/dc387565-3fa1-4271-a12e-dda0c699e3e4-kube-api-access-6zwbc\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-kubelet\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-run-openvswitch\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465425    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-sysctl-conf\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465449    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0c6c582-7916-4b6f-bee3-e8b5eedda893-cni-binary-copy\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d0c6c582-7916-4b6f-bee3-e8b5eedda893-multus-daemon-config\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465494    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-run-multus-certs\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd02445c-74cd-43da-9315-6e6cf4e5050f-konnectivity-ca\") pod \"konnectivity-agent-nmzpg\" (UID: \"dd02445c-74cd-43da-9315-6e6cf4e5050f\") " pod="kube-system/konnectivity-agent-nmzpg"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465553    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-var-lib-kubelet\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.466225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-log-socket\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465602    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-var-lib-openvswitch\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c34fc4a6-58be-48e7-83db-1a0d5619031e-ovn-node-metrics-cert\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465649    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-run-netns\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465672    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-hostroot\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465697    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465723    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-cni-bin\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465736    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-systemd-units\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mxxw\" (UniqueName: \"kubernetes.io/projected/f7c2b6aa-6cb9-4b5b-885c-4be0975036ff-kube-api-access-6mxxw\") pod \"node-ca-pgvs9\" (UID: \"f7c2b6aa-6cb9-4b5b-885c-4be0975036ff\") " pod="openshift-image-registry/node-ca-pgvs9"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465785    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-cni-binary-copy\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2657b224-c52c-4e1e-837a-8b5b48f0e035-iptables-alerter-script\") pod \"iptables-alerter-2bmr8\" (UID: \"2657b224-c52c-4e1e-837a-8b5b48f0e035\") " pod="openshift-network-operator/iptables-alerter-2bmr8"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465907    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-kubelet\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465942    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-run-openvswitch\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.465984    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-log-socket\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-etc-openvswitch\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-cni-netd\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466074    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd02445c-74cd-43da-9315-6e6cf4e5050f-agent-certs\") pod \"konnectivity-agent-nmzpg\" (UID: \"dd02445c-74cd-43da-9315-6e6cf4e5050f\") " pod="kube-system/konnectivity-agent-nmzpg"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466106    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-852gd\" (UniqueName: \"kubernetes.io/projected/e6dde057-4214-4246-bea1-a0820ab52a59-kube-api-access-852gd\") pod \"node-resolver-lczx4\" (UID: \"e6dde057-4214-4246-bea1-a0820ab52a59\") " pod="openshift-dns/node-resolver-lczx4"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466132    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466159    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466181    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466565    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-cni-bin\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466608    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466641    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-cni-netd\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466666    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-tuned\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466679    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-etc-openvswitch\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466664    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-var-lib-openvswitch\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-cnibin\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.467710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-multus-socket-dir-parent\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466762    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466821    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466826    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-registration-dir\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466871    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-registration-dir\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-sysconfig\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7c2b6aa-6cb9-4b5b-885c-4be0975036ff-serviceca\") pod \"node-ca-pgvs9\" (UID: \"f7c2b6aa-6cb9-4b5b-885c-4be0975036ff\") " pod="openshift-image-registry/node-ca-pgvs9"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.466983    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2657b224-c52c-4e1e-837a-8b5b48f0e035-host-slash\") pod \"iptables-alerter-2bmr8\" (UID: \"2657b224-c52c-4e1e-837a-8b5b48f0e035\") " pod="openshift-network-operator/iptables-alerter-2bmr8"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c34fc4a6-58be-48e7-83db-1a0d5619031e-ovnkube-script-lib\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467053    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96435aa1-34e7-463c-bbf2-79cc9d210c09-tmp\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467167    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-multus-cni-dir\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467203    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-var-lib-kubelet\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467227    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-cnibin\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467271    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c34fc4a6-58be-48e7-83db-1a0d5619031e-env-overrides\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.468408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w958t\" (UniqueName: \"kubernetes.io/projected/c34fc4a6-58be-48e7-83db-1a0d5619031e-kube-api-access-w958t\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-os-release\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467374    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7c2b6aa-6cb9-4b5b-885c-4be0975036ff-host\") pod \"node-ca-pgvs9\" (UID: \"f7c2b6aa-6cb9-4b5b-885c-4be0975036ff\") " pod="openshift-image-registry/node-ca-pgvs9"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467403    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-socket-dir\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-slash\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467448    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1da27b61-6566-4583-a851-61ea1f1e8d11-dbus\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467476    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-kubernetes\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467498    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cx5j\" (UniqueName: \"kubernetes.io/projected/96435aa1-34e7-463c-bbf2-79cc9d210c09-kube-api-access-8cx5j\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467521    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-system-cni-dir\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467545    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-run-ovn\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467567    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-node-log\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-etc-selinux\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467613    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xkrd\" (UniqueName: \"kubernetes.io/projected/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-kube-api-access-7xkrd\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467639    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:14.469047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467626    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-socket-dir\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-sysctl-d\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467670    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c34fc4a6-58be-48e7-83db-1a0d5619031e-ovnkube-script-lib\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467694    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c34fc4a6-58be-48e7-83db-1a0d5619031e-env-overrides\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-sys\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467718    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-slash\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467730    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-lib-modules\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467754    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-multus-conf-dir\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467814    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-node-log\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.467834    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467856    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-run-ovn\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467856    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-etc-selinux\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.467908    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs podName:dc387565-3fa1-4271-a12e-dda0c699e3e4 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:14.967872282 +0000 UTC m=+2.951345958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs") pod "network-metrics-daemon-v7gwd" (UID: "dc387565-3fa1-4271-a12e-dda0c699e3e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467926    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgqt\" (UniqueName: \"kubernetes.io/projected/d0c6c582-7916-4b6f-bee3-e8b5eedda893-kube-api-access-khgqt\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.469726 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467952    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1da27b61-6566-4583-a851-61ea1f1e8d11-kubelet-config\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.467975    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6bs\" (UniqueName: \"kubernetes.io/projected/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-kube-api-access-kt6bs\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-sys-fs\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-run-netns\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468078    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-sys-fs\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468089    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-run-k8s-cni-cncf-io\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468105    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c34fc4a6-58be-48e7-83db-1a0d5619031e-ovnkube-config\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468127    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-run-netns\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-modprobe-d\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468172    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-run\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-var-lib-cni-bin\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468223    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-etc-kubernetes\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468245    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6dde057-4214-4246-bea1-a0820ab52a59-hosts-file\") pod \"node-resolver-lczx4\" (UID: \"e6dde057-4214-4246-bea1-a0820ab52a59\") " pod="openshift-dns/node-resolver-lczx4"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468269    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlmk\" (UniqueName: \"kubernetes.io/projected/2657b224-c52c-4e1e-837a-8b5b48f0e035-kube-api-access-qxlmk\") pod \"iptables-alerter-2bmr8\" (UID: \"2657b224-c52c-4e1e-837a-8b5b48f0e035\") " pod="openshift-network-operator/iptables-alerter-2bmr8"
Jul 02 12:39:14.470470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468313    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-device-dir\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-run-systemd\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468361    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-host\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468386    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6dde057-4214-4246-bea1-a0820ab52a59-tmp-dir\") pod \"node-resolver-lczx4\" (UID: \"e6dde057-4214-4246-bea1-a0820ab52a59\") " pod="openshift-dns/node-resolver-lczx4"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-system-cni-dir\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468460    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-run-ovn-kubernetes\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468482    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-systemd\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468507    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468549    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c34fc4a6-58be-48e7-83db-1a0d5619031e-ovnkube-config\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468619    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-run-systemd\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468631    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c34fc4a6-58be-48e7-83db-1a0d5619031e-host-run-ovn-kubernetes\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.468688    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-device-dir\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.471111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.470426    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c34fc4a6-58be-48e7-83db-1a0d5619031e-ovn-node-metrics-cert\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.481728 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.481706    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:39:14.481728 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.481727    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:39:14.481858 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.481738    2564 projected.go:196] Error preparing data for projected volume kube-api-access-k2zck for pod openshift-network-diagnostics/network-check-target-r975p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:14.481858 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.481790    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck podName:2697b9b4-7477-4e35-a9f3-298ed7a423ed nodeName:}" failed. No retries permitted until 2026-07-02 12:39:14.981777316 +0000 UTC m=+2.965250977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k2zck" (UniqueName: "kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck") pod "network-check-target-r975p" (UID: "2697b9b4-7477-4e35-a9f3-298ed7a423ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:14.484615 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.484546    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zwbc\" (UniqueName: \"kubernetes.io/projected/dc387565-3fa1-4271-a12e-dda0c699e3e4-kube-api-access-6zwbc\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:14.485503 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.485477    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w958t\" (UniqueName: \"kubernetes.io/projected/c34fc4a6-58be-48e7-83db-1a0d5619031e-kube-api-access-w958t\") pod \"ovnkube-node-qmx4p\" (UID: \"c34fc4a6-58be-48e7-83db-1a0d5619031e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.489776 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.489755    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xkrd\" (UniqueName: \"kubernetes.io/projected/ddb014eb-46fe-43cf-bcdc-e7dcfc07993c-kube-api-access-7xkrd\") pod \"aws-ebs-csi-driver-node-g8z57\" (UID: \"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.568872 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.568783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.568872 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.568824    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-systemd\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.568872 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.568842    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.569152 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.568996    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-systemd\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.569152 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569016    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.569152 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569094    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-var-lib-cni-multus\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.569152 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569116    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-os-release\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.569152 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-sysctl-conf\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.569397 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0c6c582-7916-4b6f-bee3-e8b5eedda893-cni-binary-copy\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.569397 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569197    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d0c6c582-7916-4b6f-bee3-e8b5eedda893-multus-daemon-config\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.569397 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-run-multus-certs\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.569397 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569272    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd02445c-74cd-43da-9315-6e6cf4e5050f-konnectivity-ca\") pod \"konnectivity-agent-nmzpg\" (UID: \"dd02445c-74cd-43da-9315-6e6cf4e5050f\") " pod="kube-system/konnectivity-agent-nmzpg"
Jul 02 12:39:14.569397 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569311    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-var-lib-kubelet\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.569397 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-run-netns\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.569397 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-hostroot\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.569397 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mxxw\" (UniqueName: \"kubernetes.io/projected/f7c2b6aa-6cb9-4b5b-885c-4be0975036ff-kube-api-access-6mxxw\") pod \"node-ca-pgvs9\" (UID: \"f7c2b6aa-6cb9-4b5b-885c-4be0975036ff\") " pod="openshift-image-registry/node-ca-pgvs9"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-cni-binary-copy\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569456    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2657b224-c52c-4e1e-837a-8b5b48f0e035-iptables-alerter-script\") pod \"iptables-alerter-2bmr8\" (UID: \"2657b224-c52c-4e1e-837a-8b5b48f0e035\") " pod="openshift-network-operator/iptables-alerter-2bmr8"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd02445c-74cd-43da-9315-6e6cf4e5050f-agent-certs\") pod \"konnectivity-agent-nmzpg\" (UID: \"dd02445c-74cd-43da-9315-6e6cf4e5050f\") " pod="kube-system/konnectivity-agent-nmzpg"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569506    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852gd\" (UniqueName: \"kubernetes.io/projected/e6dde057-4214-4246-bea1-a0820ab52a59-kube-api-access-852gd\") pod \"node-resolver-lczx4\" (UID: \"e6dde057-4214-4246-bea1-a0820ab52a59\") " pod="openshift-dns/node-resolver-lczx4"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569554    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-tuned\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569559    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569576    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-cnibin\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569600    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-multus-socket-dir-parent\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-sysconfig\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569646    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7c2b6aa-6cb9-4b5b-885c-4be0975036ff-serviceca\") pod \"node-ca-pgvs9\" (UID: \"f7c2b6aa-6cb9-4b5b-885c-4be0975036ff\") " pod="openshift-image-registry/node-ca-pgvs9"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2657b224-c52c-4e1e-837a-8b5b48f0e035-host-slash\") pod \"iptables-alerter-2bmr8\" (UID: \"2657b224-c52c-4e1e-837a-8b5b48f0e035\") " pod="openshift-network-operator/iptables-alerter-2bmr8"
Jul 02 12:39:14.569714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569694    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0c6c582-7916-4b6f-bee3-e8b5eedda893-cni-binary-copy\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569701    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-sysctl-conf\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569758    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-os-release\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96435aa1-34e7-463c-bbf2-79cc9d210c09-tmp\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-multus-cni-dir\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569807    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-var-lib-kubelet\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-cnibin\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-os-release\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569870    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7c2b6aa-6cb9-4b5b-885c-4be0975036ff-host\") pod \"node-ca-pgvs9\" (UID: \"f7c2b6aa-6cb9-4b5b-885c-4be0975036ff\") " pod="openshift-image-registry/node-ca-pgvs9"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1da27b61-6566-4583-a851-61ea1f1e8d11-dbus\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569916    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-kubernetes\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cx5j\" (UniqueName: \"kubernetes.io/projected/96435aa1-34e7-463c-bbf2-79cc9d210c09-kube-api-access-8cx5j\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569950    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-system-cni-dir\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.569979    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-sysctl-d\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-sys\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.570266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-lib-modules\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570066    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-multus-conf-dir\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgqt\" (UniqueName: \"kubernetes.io/projected/d0c6c582-7916-4b6f-bee3-e8b5eedda893-kube-api-access-khgqt\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570106    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1da27b61-6566-4583-a851-61ea1f1e8d11-kubelet-config\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6bs\" (UniqueName: \"kubernetes.io/projected/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-kube-api-access-kt6bs\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570134    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d0c6c582-7916-4b6f-bee3-e8b5eedda893-multus-daemon-config\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570181    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-kubernetes\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570188    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-run-k8s-cni-cncf-io\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570204    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-multus-cni-dir\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570240    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7c2b6aa-6cb9-4b5b-885c-4be0975036ff-host\") pod \"node-ca-pgvs9\" (UID: \"f7c2b6aa-6cb9-4b5b-885c-4be0975036ff\") " pod="openshift-image-registry/node-ca-pgvs9"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570272    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-multus-conf-dir\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570327    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-sysctl-d\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570346    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1da27b61-6566-4583-a851-61ea1f1e8d11-dbus\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570353    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-sys\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570369    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-var-lib-kubelet\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571005 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570413    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-cnibin\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570453    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-lib-modules\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570468    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-os-release\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570511    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1da27b61-6566-4583-a851-61ea1f1e8d11-kubelet-config\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570549    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2657b224-c52c-4e1e-837a-8b5b48f0e035-iptables-alerter-script\") pod \"iptables-alerter-2bmr8\" (UID: \"2657b224-c52c-4e1e-837a-8b5b48f0e035\") " pod="openshift-network-operator/iptables-alerter-2bmr8"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570616    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-run-multus-certs\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570643    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-sysconfig\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570663    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570686    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-run-netns\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570701    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-cnibin\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570721    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-system-cni-dir\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570743    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-multus-socket-dir-parent\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570789    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2657b224-c52c-4e1e-837a-8b5b48f0e035-host-slash\") pod \"iptables-alerter-2bmr8\" (UID: \"2657b224-c52c-4e1e-837a-8b5b48f0e035\") " pod="openshift-network-operator/iptables-alerter-2bmr8"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-run-k8s-cni-cncf-io\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570826    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-modprobe-d\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.571748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-run\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-var-lib-cni-bin\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-etc-kubernetes\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6dde057-4214-4246-bea1-a0820ab52a59-hosts-file\") pod \"node-resolver-lczx4\" (UID: \"e6dde057-4214-4246-bea1-a0820ab52a59\") " pod="openshift-dns/node-resolver-lczx4"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570955    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlmk\" (UniqueName: \"kubernetes.io/projected/2657b224-c52c-4e1e-837a-8b5b48f0e035-kube-api-access-qxlmk\") pod \"iptables-alerter-2bmr8\" (UID: \"2657b224-c52c-4e1e-837a-8b5b48f0e035\") " pod="openshift-network-operator/iptables-alerter-2bmr8"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.570982    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-host\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6dde057-4214-4246-bea1-a0820ab52a59-tmp-dir\") pod \"node-resolver-lczx4\" (UID: \"e6dde057-4214-4246-bea1-a0820ab52a59\") " pod="openshift-dns/node-resolver-lczx4"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571023    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-system-cni-dir\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571077    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-system-cni-dir\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571134    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd02445c-74cd-43da-9315-6e6cf4e5050f-konnectivity-ca\") pod \"konnectivity-agent-nmzpg\" (UID: \"dd02445c-74cd-43da-9315-6e6cf4e5050f\") " pod="kube-system/konnectivity-agent-nmzpg"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571149    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-modprobe-d\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571139    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7c2b6aa-6cb9-4b5b-885c-4be0975036ff-serviceca\") pod \"node-ca-pgvs9\" (UID: \"f7c2b6aa-6cb9-4b5b-885c-4be0975036ff\") " pod="openshift-image-registry/node-ca-pgvs9"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571183    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-var-lib-kubelet\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571186    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-run\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571215    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-etc-kubernetes\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.572342 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.571231    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:14.572882 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571251    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96435aa1-34e7-463c-bbf2-79cc9d210c09-host\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.572882 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.571286    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret podName:1da27b61-6566-4583-a851-61ea1f1e8d11 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:15.071269238 +0000 UTC m=+3.054742912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret") pod "global-pull-secret-syncer-bx9gl" (UID: "1da27b61-6566-4583-a851-61ea1f1e8d11") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:14.572882 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571378    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6dde057-4214-4246-bea1-a0820ab52a59-hosts-file\") pod \"node-resolver-lczx4\" (UID: \"e6dde057-4214-4246-bea1-a0820ab52a59\") " pod="openshift-dns/node-resolver-lczx4"
Jul 02 12:39:14.572882 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571434    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-var-lib-cni-bin\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.572882 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571472    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-hostroot\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.572882 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571481    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6dde057-4214-4246-bea1-a0820ab52a59-tmp-dir\") pod \"node-resolver-lczx4\" (UID: \"e6dde057-4214-4246-bea1-a0820ab52a59\") " pod="openshift-dns/node-resolver-lczx4"
Jul 02 12:39:14.572882 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571599    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d0c6c582-7916-4b6f-bee3-e8b5eedda893-host-var-lib-cni-multus\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.572882 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.571768    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-cni-binary-copy\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.572882 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.572409    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96435aa1-34e7-463c-bbf2-79cc9d210c09-tmp\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.573342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.573279    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd02445c-74cd-43da-9315-6e6cf4e5050f-agent-certs\") pod \"konnectivity-agent-nmzpg\" (UID: \"dd02445c-74cd-43da-9315-6e6cf4e5050f\") " pod="kube-system/konnectivity-agent-nmzpg"
Jul 02 12:39:14.573579 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.573554    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/96435aa1-34e7-463c-bbf2-79cc9d210c09-etc-tuned\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.582607 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.582577    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-852gd\" (UniqueName: \"kubernetes.io/projected/e6dde057-4214-4246-bea1-a0820ab52a59-kube-api-access-852gd\") pod \"node-resolver-lczx4\" (UID: \"e6dde057-4214-4246-bea1-a0820ab52a59\") " pod="openshift-dns/node-resolver-lczx4"
Jul 02 12:39:14.582857 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.582838    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cx5j\" (UniqueName: \"kubernetes.io/projected/96435aa1-34e7-463c-bbf2-79cc9d210c09-kube-api-access-8cx5j\") pod \"tuned-xhsqd\" (UID: \"96435aa1-34e7-463c-bbf2-79cc9d210c09\") " pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.586314 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.586250    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6bs\" (UniqueName: \"kubernetes.io/projected/cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40-kube-api-access-kt6bs\") pod \"multus-additional-cni-plugins-j2cwd\" (UID: \"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40\") " pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.586414 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.586377    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgqt\" (UniqueName: \"kubernetes.io/projected/d0c6c582-7916-4b6f-bee3-e8b5eedda893-kube-api-access-khgqt\") pod \"multus-sgqpb\" (UID: \"d0c6c582-7916-4b6f-bee3-e8b5eedda893\") " pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.587632 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.587613    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mxxw\" (UniqueName: \"kubernetes.io/projected/f7c2b6aa-6cb9-4b5b-885c-4be0975036ff-kube-api-access-6mxxw\") pod \"node-ca-pgvs9\" (UID: \"f7c2b6aa-6cb9-4b5b-885c-4be0975036ff\") " pod="openshift-image-registry/node-ca-pgvs9"
Jul 02 12:39:14.596691 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.596671    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlmk\" (UniqueName: \"kubernetes.io/projected/2657b224-c52c-4e1e-837a-8b5b48f0e035-kube-api-access-qxlmk\") pod \"iptables-alerter-2bmr8\" (UID: \"2657b224-c52c-4e1e-837a-8b5b48f0e035\") " pod="openshift-network-operator/iptables-alerter-2bmr8"
Jul 02 12:39:14.654018 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.653994    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57"
Jul 02 12:39:14.666441 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.666420    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:14.676623 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.676606    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nmzpg"
Jul 02 12:39:14.683615 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.683598    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xhsqd"
Jul 02 12:39:14.692386 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.692365    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lczx4"
Jul 02 12:39:14.700876 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.700860    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j2cwd"
Jul 02 12:39:14.708125 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.708109    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sgqpb"
Jul 02 12:39:14.719768 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.719749    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pgvs9"
Jul 02 12:39:14.728363 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.728319    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2bmr8"
Jul 02 12:39:14.973158 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:14.973082    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:14.973340 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.973217    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:14.973340 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:14.973290    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs podName:dc387565-3fa1-4271-a12e-dda0c699e3e4 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:15.973268616 +0000 UTC m=+3.956742283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs") pod "network-metrics-daemon-v7gwd" (UID: "dc387565-3fa1-4271-a12e-dda0c699e3e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:15.074371 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:15.074333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:15.074540 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:15.074418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zck\" (UniqueName: \"kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck\") pod \"network-check-target-r975p\" (UID: \"2697b9b4-7477-4e35-a9f3-298ed7a423ed\") " pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:15.074540 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:15.074509    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:15.074540 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:15.074539    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:39:15.074705 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:15.074554    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:39:15.074705 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:15.074564    2564 projected.go:196] Error preparing data for projected volume kube-api-access-k2zck for pod openshift-network-diagnostics/network-check-target-r975p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:15.074705 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:15.074591    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret podName:1da27b61-6566-4583-a851-61ea1f1e8d11 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:16.074571816 +0000 UTC m=+4.058045480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret") pod "global-pull-secret-syncer-bx9gl" (UID: "1da27b61-6566-4583-a851-61ea1f1e8d11") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:15.074705 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:15.074609    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck podName:2697b9b4-7477-4e35-a9f3-298ed7a423ed nodeName:}" failed. No retries permitted until 2026-07-02 12:39:16.074597265 +0000 UTC m=+4.058070926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2zck" (UniqueName: "kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck") pod "network-check-target-r975p" (UID: "2697b9b4-7477-4e35-a9f3-298ed7a423ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:15.440665 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:15.440567    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 12:34:14 +0000 UTC" deadline="2028-04-09 06:13:44.320632858 +0000 UTC"
Jul 02 12:39:15.440665 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:15.440602    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15521h34m28.88003624s"
Jul 02 12:39:15.582625 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:15.582594    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb014eb_46fe_43cf_bcdc_e7dcfc07993c.slice/crio-aa942e1ba556d9ef1f93b5fdec62898f87a3ab7f79bcb8066447b93478284c66 WatchSource:0}: Error finding container aa942e1ba556d9ef1f93b5fdec62898f87a3ab7f79bcb8066447b93478284c66: Status 404 returned error can't find the container with id aa942e1ba556d9ef1f93b5fdec62898f87a3ab7f79bcb8066447b93478284c66
Jul 02 12:39:15.584260 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:15.584222    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6dde057_4214_4246_bea1_a0820ab52a59.slice/crio-2408d0594691413e09cbb750c3906bdbe7bb1d9f01f2bce343dbb35c2d76cba2 WatchSource:0}: Error finding container 2408d0594691413e09cbb750c3906bdbe7bb1d9f01f2bce343dbb35c2d76cba2: Status 404 returned error can't find the container with id 2408d0594691413e09cbb750c3906bdbe7bb1d9f01f2bce343dbb35c2d76cba2
Jul 02 12:39:15.587434 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:15.587416    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3c40cd_00cb_4a1b_82a8_dfde87d1fd40.slice/crio-ea8f28c87e66801617e275bf05af4af7b0ea4205d9789d0c37ded4004880763e WatchSource:0}: Error finding container ea8f28c87e66801617e275bf05af4af7b0ea4205d9789d0c37ded4004880763e: Status 404 returned error can't find the container with id ea8f28c87e66801617e275bf05af4af7b0ea4205d9789d0c37ded4004880763e
Jul 02 12:39:15.589064 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:15.588971    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c6c582_7916_4b6f_bee3_e8b5eedda893.slice/crio-222fffe23ddbfafcfa5a6f2dd8d853074a025bad4f93dd6ca8b1e6e9e5d86c94 WatchSource:0}: Error finding container 222fffe23ddbfafcfa5a6f2dd8d853074a025bad4f93dd6ca8b1e6e9e5d86c94: Status 404 returned error can't find the container with id 222fffe23ddbfafcfa5a6f2dd8d853074a025bad4f93dd6ca8b1e6e9e5d86c94
Jul 02 12:39:15.589776 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:15.589757    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd02445c_74cd_43da_9315_6e6cf4e5050f.slice/crio-eac7f4ec4e460f98a57458de6eaed5a1579064266d7f938052a1aaf1ad10bfbd WatchSource:0}: Error finding container eac7f4ec4e460f98a57458de6eaed5a1579064266d7f938052a1aaf1ad10bfbd: Status 404 returned error can't find the container with id eac7f4ec4e460f98a57458de6eaed5a1579064266d7f938052a1aaf1ad10bfbd
Jul 02 12:39:15.591669 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:15.591647    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2657b224_c52c_4e1e_837a_8b5b48f0e035.slice/crio-93979850d59bf6c742b5822bfbf3dd3f991c7d436eec8b60f3828602c9c86008 WatchSource:0}: Error finding container 93979850d59bf6c742b5822bfbf3dd3f991c7d436eec8b60f3828602c9c86008: Status 404 returned error can't find the container with id 93979850d59bf6c742b5822bfbf3dd3f991c7d436eec8b60f3828602c9c86008
Jul 02 12:39:15.593631 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:15.593277    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c2b6aa_6cb9_4b5b_885c_4be0975036ff.slice/crio-8d4097e626669951fce9781e3b1bc896d87a88dffcfd4ac4f8d1057c034420d0 WatchSource:0}: Error finding container 8d4097e626669951fce9781e3b1bc896d87a88dffcfd4ac4f8d1057c034420d0: Status 404 returned error can't find the container with id 8d4097e626669951fce9781e3b1bc896d87a88dffcfd4ac4f8d1057c034420d0
Jul 02 12:39:15.593998 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:15.593855    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34fc4a6_58be_48e7_83db_1a0d5619031e.slice/crio-96ce1dce63a119fa2b775c35f256eb41585af5bd114640ac72699caeff14ae80 WatchSource:0}: Error finding container 96ce1dce63a119fa2b775c35f256eb41585af5bd114640ac72699caeff14ae80: Status 404 returned error can't find the container with id 96ce1dce63a119fa2b775c35f256eb41585af5bd114640ac72699caeff14ae80
Jul 02 12:39:15.594764 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:15.594737    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96435aa1_34e7_463c_bbf2_79cc9d210c09.slice/crio-ff29947f571996f976491c7d80ec66c36489ef5244ff74cd8d9b447004b33a20 WatchSource:0}: Error finding container ff29947f571996f976491c7d80ec66c36489ef5244ff74cd8d9b447004b33a20: Status 404 returned error can't find the container with id ff29947f571996f976491c7d80ec66c36489ef5244ff74cd8d9b447004b33a20
Jul 02 12:39:15.982359 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:15.981936    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:15.982537 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:15.982100    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:15.982595 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:15.982576    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs podName:dc387565-3fa1-4271-a12e-dda0c699e3e4 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:17.982553269 +0000 UTC m=+5.966026932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs") pod "network-metrics-daemon-v7gwd" (UID: "dc387565-3fa1-4271-a12e-dda0c699e3e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:16.083117 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.082780    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zck\" (UniqueName: \"kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck\") pod \"network-check-target-r975p\" (UID: \"2697b9b4-7477-4e35-a9f3-298ed7a423ed\") " pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:16.083117 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.082834    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:16.083117 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:16.082953    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:16.083117 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:16.082982    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:39:16.083117 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:16.082999    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:39:16.083117 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:16.083010    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret podName:1da27b61-6566-4583-a851-61ea1f1e8d11 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:18.082992045 +0000 UTC m=+6.066465707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret") pod "global-pull-secret-syncer-bx9gl" (UID: "1da27b61-6566-4583-a851-61ea1f1e8d11") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:16.083117 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:16.083011    2564 projected.go:196] Error preparing data for projected volume kube-api-access-k2zck for pod openshift-network-diagnostics/network-check-target-r975p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:16.083117 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:16.083046    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck podName:2697b9b4-7477-4e35-a9f3-298ed7a423ed nodeName:}" failed. No retries permitted until 2026-07-02 12:39:18.083037167 +0000 UTC m=+6.066510828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2zck" (UniqueName: "kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck") pod "network-check-target-r975p" (UID: "2697b9b4-7477-4e35-a9f3-298ed7a423ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:16.421418 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.420617    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:16.421418 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:16.420743    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:16.421418 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.421126    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:16.421418 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:16.421237    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:16.423146 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.423002    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:16.423146 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:16.423106    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:16.441678 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.441602    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 12:34:14 +0000 UTC" deadline="2028-03-03 08:42:04.880390382 +0000 UTC"
Jul 02 12:39:16.441678 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.441655    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14636h2m48.438738593s"
Jul 02 12:39:16.455705 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.455559    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p" event={"ID":"c34fc4a6-58be-48e7-83db-1a0d5619031e","Type":"ContainerStarted","Data":"96ce1dce63a119fa2b775c35f256eb41585af5bd114640ac72699caeff14ae80"}
Jul 02 12:39:16.481351 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.470124    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sgqpb" event={"ID":"d0c6c582-7916-4b6f-bee3-e8b5eedda893","Type":"ContainerStarted","Data":"222fffe23ddbfafcfa5a6f2dd8d853074a025bad4f93dd6ca8b1e6e9e5d86c94"}
Jul 02 12:39:16.481351 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.473893    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lczx4" event={"ID":"e6dde057-4214-4246-bea1-a0820ab52a59","Type":"ContainerStarted","Data":"2408d0594691413e09cbb750c3906bdbe7bb1d9f01f2bce343dbb35c2d76cba2"}
Jul 02 12:39:16.481351 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.477521    2564 generic.go:356] "Generic (PLEG): container finished" podID="f3966a64f6cf985e92da6b4f85445732" containerID="0a44e90d1506442ad7d3fac896807de7447594f59badeaa5f2a77afe4230e564" exitCode=0
Jul 02 12:39:16.481351 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.477629    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal" event={"ID":"f3966a64f6cf985e92da6b4f85445732","Type":"ContainerDied","Data":"0a44e90d1506442ad7d3fac896807de7447594f59badeaa5f2a77afe4230e564"}
Jul 02 12:39:16.485493 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.485448    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal" event={"ID":"6a9ee241c13dace2d3c74948d3ea9c6f","Type":"ContainerStarted","Data":"720f0b81cef698196c31dd10f8f1c5fb5daac6aaa9117e27f35d100fbb81d15b"}
Jul 02 12:39:16.501560 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.501534    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pgvs9" event={"ID":"f7c2b6aa-6cb9-4b5b-885c-4be0975036ff","Type":"ContainerStarted","Data":"8d4097e626669951fce9781e3b1bc896d87a88dffcfd4ac4f8d1057c034420d0"}
Jul 02 12:39:16.504660 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.504634    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nmzpg" event={"ID":"dd02445c-74cd-43da-9315-6e6cf4e5050f","Type":"ContainerStarted","Data":"eac7f4ec4e460f98a57458de6eaed5a1579064266d7f938052a1aaf1ad10bfbd"}
Jul 02 12:39:16.506700 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.506676    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2bmr8" event={"ID":"2657b224-c52c-4e1e-837a-8b5b48f0e035","Type":"ContainerStarted","Data":"93979850d59bf6c742b5822bfbf3dd3f991c7d436eec8b60f3828602c9c86008"}
Jul 02 12:39:16.515174 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.515151    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2cwd" event={"ID":"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40","Type":"ContainerStarted","Data":"ea8f28c87e66801617e275bf05af4af7b0ea4205d9789d0c37ded4004880763e"}
Jul 02 12:39:16.526988 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.526923    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57" event={"ID":"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c","Type":"ContainerStarted","Data":"aa942e1ba556d9ef1f93b5fdec62898f87a3ab7f79bcb8066447b93478284c66"}
Jul 02 12:39:16.532187 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:16.532116    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xhsqd" event={"ID":"96435aa1-34e7-463c-bbf2-79cc9d210c09","Type":"ContainerStarted","Data":"ff29947f571996f976491c7d80ec66c36489ef5244ff74cd8d9b447004b33a20"}
Jul 02 12:39:17.554961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:17.554886    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal" event={"ID":"f3966a64f6cf985e92da6b4f85445732","Type":"ContainerStarted","Data":"a25f71f28aae02a4bef0a00272c48d969fc68dc279a9b9cd7ca841f1c2886b4e"}
Jul 02 12:39:17.573824 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:17.572840    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal" podStartSLOduration=4.572819733 podStartE2EDuration="4.572819733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:16.511443461 +0000 UTC m=+4.494917141" watchObservedRunningTime="2026-07-02 12:39:17.572819733 +0000 UTC m=+5.556293417"
Jul 02 12:39:17.999397 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:17.998710    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:17.999397 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:17.998938    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:17.999397 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:17.999000    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs podName:dc387565-3fa1-4271-a12e-dda0c699e3e4 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:21.998981727 +0000 UTC m=+9.982455391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs") pod "network-metrics-daemon-v7gwd" (UID: "dc387565-3fa1-4271-a12e-dda0c699e3e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:18.099501 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:18.099463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zck\" (UniqueName: \"kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck\") pod \"network-check-target-r975p\" (UID: \"2697b9b4-7477-4e35-a9f3-298ed7a423ed\") " pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:18.099660 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:18.099528    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:18.099741 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:18.099715    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:18.099792 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:18.099774    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret podName:1da27b61-6566-4583-a851-61ea1f1e8d11 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:22.099757208 +0000 UTC m=+10.083230874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret") pod "global-pull-secret-syncer-bx9gl" (UID: "1da27b61-6566-4583-a851-61ea1f1e8d11") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:18.100196 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:18.100174    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:39:18.100196 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:18.100199    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:39:18.100362 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:18.100212    2564 projected.go:196] Error preparing data for projected volume kube-api-access-k2zck for pod openshift-network-diagnostics/network-check-target-r975p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:18.100362 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:18.100291    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck podName:2697b9b4-7477-4e35-a9f3-298ed7a423ed nodeName:}" failed. No retries permitted until 2026-07-02 12:39:22.100244714 +0000 UTC m=+10.083718392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2zck" (UniqueName: "kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck") pod "network-check-target-r975p" (UID: "2697b9b4-7477-4e35-a9f3-298ed7a423ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:18.420261 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:18.419915    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:18.420261 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:18.420037    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:18.420261 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:18.420092    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:18.429956 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:18.420886    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:18.429956 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:18.429742    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:18.429956 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:18.429857    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:20.417058 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:20.417028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:20.417624 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:20.417166    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:20.417624 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:20.417546    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:20.417736 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:20.417638    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:20.417736 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:20.417665    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:20.417852 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:20.417738    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:22.030600 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:22.030559    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:22.030994 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:22.030697    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:22.030994 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:22.030751    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs podName:dc387565-3fa1-4271-a12e-dda0c699e3e4 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:30.030737155 +0000 UTC m=+18.014210819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs") pod "network-metrics-daemon-v7gwd" (UID: "dc387565-3fa1-4271-a12e-dda0c699e3e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:22.132576 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:22.131741    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zck\" (UniqueName: \"kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck\") pod \"network-check-target-r975p\" (UID: \"2697b9b4-7477-4e35-a9f3-298ed7a423ed\") " pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:22.132576 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:22.131804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:22.132576 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:22.131955    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:22.132576 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:22.132016    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret podName:1da27b61-6566-4583-a851-61ea1f1e8d11 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:30.131998318 +0000 UTC m=+18.115471986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret") pod "global-pull-secret-syncer-bx9gl" (UID: "1da27b61-6566-4583-a851-61ea1f1e8d11") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:22.132576 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:22.132455    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:39:22.132576 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:22.132475    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:39:22.132576 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:22.132488    2564 projected.go:196] Error preparing data for projected volume kube-api-access-k2zck for pod openshift-network-diagnostics/network-check-target-r975p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:22.132576 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:22.132539    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck podName:2697b9b4-7477-4e35-a9f3-298ed7a423ed nodeName:}" failed. No retries permitted until 2026-07-02 12:39:30.132523565 +0000 UTC m=+18.115997227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2zck" (UniqueName: "kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck") pod "network-check-target-r975p" (UID: "2697b9b4-7477-4e35-a9f3-298ed7a423ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:22.418603 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:22.417816    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:22.418603 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:22.417949    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:22.418603 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:22.418290    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:22.418603 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:22.418410    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:22.418603 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:22.418459    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:22.418603 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:22.418525    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:24.417936 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:24.417887    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:24.418500 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:24.418025    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:24.418500 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:24.418067    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:24.418500 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:24.418180    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:24.418500 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:24.418242    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:24.418500 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:24.418359    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:26.417217 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:26.417123    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:26.417651 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:26.417123    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:26.417651 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:26.417279    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:26.417651 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:26.417123    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:26.417651 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:26.417361    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:26.417651 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:26.417418    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:28.417031 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:28.416997    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:28.417031 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:28.417013    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:28.417536 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:28.417003    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:28.417536 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:28.417116    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:28.417536 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:28.417212    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:28.417536 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:28.417327    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:30.096166 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:30.096130    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:30.096609 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:30.096282    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:30.096609 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:30.096361    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs podName:dc387565-3fa1-4271-a12e-dda0c699e3e4 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:46.096343003 +0000 UTC m=+34.079816672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs") pod "network-metrics-daemon-v7gwd" (UID: "dc387565-3fa1-4271-a12e-dda0c699e3e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:30.197509 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:30.197476    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zck\" (UniqueName: \"kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck\") pod \"network-check-target-r975p\" (UID: \"2697b9b4-7477-4e35-a9f3-298ed7a423ed\") " pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:30.197679 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:30.197523    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:30.197679 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:30.197609    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:30.197679 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:30.197644    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:39:30.197679 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:30.197663    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret podName:1da27b61-6566-4583-a851-61ea1f1e8d11 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:46.197645151 +0000 UTC m=+34.181118812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret") pod "global-pull-secret-syncer-bx9gl" (UID: "1da27b61-6566-4583-a851-61ea1f1e8d11") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:30.197679 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:30.197670    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:39:30.197890 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:30.197685    2564 projected.go:196] Error preparing data for projected volume kube-api-access-k2zck for pod openshift-network-diagnostics/network-check-target-r975p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:30.197890 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:30.197733    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck podName:2697b9b4-7477-4e35-a9f3-298ed7a423ed nodeName:}" failed. No retries permitted until 2026-07-02 12:39:46.197717558 +0000 UTC m=+34.181191227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2zck" (UniqueName: "kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck") pod "network-check-target-r975p" (UID: "2697b9b4-7477-4e35-a9f3-298ed7a423ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:30.420097 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:30.420023    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:30.420252 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:30.420131    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:30.420252 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:30.420151    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:30.420252 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:30.420244    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:30.420442 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:30.420311    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:30.420442 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:30.420407    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:32.418524 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:32.418489    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:32.418985 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:32.418590    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:32.418985 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:32.418630    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:32.418985 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:32.418632    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:32.418985 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:32.418691    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:32.418985 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:32.418770    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:34.418749 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.418382    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:34.418749 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:34.418503    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:34.424331 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.421442    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:34.424331 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:34.421584    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:34.424772 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.419364    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:34.424772 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:34.424723    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:34.596231 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.595718    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pgvs9" event={"ID":"f7c2b6aa-6cb9-4b5b-885c-4be0975036ff","Type":"ContainerStarted","Data":"61232009aa3b74437320768688a15b8e7da8c7bc161d5f54ad8f8afd60259f80"}
Jul 02 12:39:34.597945 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.597638    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nmzpg" event={"ID":"dd02445c-74cd-43da-9315-6e6cf4e5050f","Type":"ContainerStarted","Data":"0beb5b6ee7d9c9eaaa9b03f81379e7503ed550ff462da44daa2112bc26f9c765"}
Jul 02 12:39:34.601486 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.600534    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2cwd" event={"ID":"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40","Type":"ContainerStarted","Data":"c5f78971a58714595b038458f3ec6850a3e5c240aa1b2d33f05869d6c1b4bedf"}
Jul 02 12:39:34.603695 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.603668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57" event={"ID":"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c","Type":"ContainerStarted","Data":"e809f3ba75491dd3139281a9200bf9451a0ec14a3423832a187cdcfa99cd3c71"}
Jul 02 12:39:34.605141 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.605107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xhsqd" event={"ID":"96435aa1-34e7-463c-bbf2-79cc9d210c09","Type":"ContainerStarted","Data":"9f4531e36c5464d67e6969125d8e1ca9e130cadecfa149f9be8611f908151339"}
Jul 02 12:39:34.607164 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.607141    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p" event={"ID":"c34fc4a6-58be-48e7-83db-1a0d5619031e","Type":"ContainerStarted","Data":"46a40fe5a10173d6e2496b3d6ea2e2d88c32bb747f7e761a7c236d58e529dcb0"}
Jul 02 12:39:34.608580 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.608561    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sgqpb" event={"ID":"d0c6c582-7916-4b6f-bee3-e8b5eedda893","Type":"ContainerStarted","Data":"72723fae28e3806e4d3bec29b4ba46ee3136b1a74301329a6dc31e1c20f6e295"}
Jul 02 12:39:34.609776 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.609758    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lczx4" event={"ID":"e6dde057-4214-4246-bea1-a0820ab52a59","Type":"ContainerStarted","Data":"20803126581f82815d0f3a6ac062dbe122c2dafc0b408b3c0beb16cfb8d28c10"}
Jul 02 12:39:34.614620 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.614582    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal" podStartSLOduration=21.614569975 podStartE2EDuration="21.614569975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:17.573807334 +0000 UTC m=+5.557281031" watchObservedRunningTime="2026-07-02 12:39:34.614569975 +0000 UTC m=+22.598043658"
Jul 02 12:39:34.629601 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.629565    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pgvs9" podStartSLOduration=3.851845826 podStartE2EDuration="22.629546538s" totalImagesPullingTime="18.777700712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:34.61436331 +0000 UTC m=+22.597836992" watchObservedRunningTime="2026-07-02 12:39:34.629546538 +0000 UTC m=+22.613020220"
Jul 02 12:39:34.630111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.630073    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nmzpg" podStartSLOduration=12.068197878 podStartE2EDuration="22.630058621s" totalImagesPullingTime="10.561860743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:34.630051594 +0000 UTC m=+22.613525281" watchObservedRunningTime="2026-07-02 12:39:34.630058621 +0000 UTC m=+22.613532308"
Jul 02 12:39:34.698570 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.698244    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xhsqd" podStartSLOduration=3.919997633 podStartE2EDuration="22.698222243s" totalImagesPullingTime="18.77822461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:34.677699515 +0000 UTC m=+22.661173198" watchObservedRunningTime="2026-07-02 12:39:34.698222243 +0000 UTC m=+22.681695927"
Jul 02 12:39:34.725840 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.725784    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lczx4" podStartSLOduration=3.950542288 podStartE2EDuration="22.725762996s" totalImagesPullingTime="18.775220708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:34.698220818 +0000 UTC m=+22.681694501" watchObservedRunningTime="2026-07-02 12:39:34.725762996 +0000 UTC m=+22.709236744"
Jul 02 12:39:34.725977 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:34.725940    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sgqpb" podStartSLOduration=3.895237831 podStartE2EDuration="22.725930739s" totalImagesPullingTime="18.830692908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:34.724825576 +0000 UTC m=+22.708299260" watchObservedRunningTime="2026-07-02 12:39:34.725930739 +0000 UTC m=+22.709404448"
Jul 02 12:39:35.612985 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:35.612957    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2bmr8" event={"ID":"2657b224-c52c-4e1e-837a-8b5b48f0e035","Type":"ContainerStarted","Data":"2791133e01c39b39509115b4ecae84b064f0c82724c64c1760d30917f5f6a371"}
Jul 02 12:39:35.614419 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:35.614395    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40" containerID="c5f78971a58714595b038458f3ec6850a3e5c240aa1b2d33f05869d6c1b4bedf" exitCode=0
Jul 02 12:39:35.614509 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:35.614485    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2cwd" event={"ID":"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40","Type":"ContainerDied","Data":"c5f78971a58714595b038458f3ec6850a3e5c240aa1b2d33f05869d6c1b4bedf"}
Jul 02 12:39:35.617623 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:35.617604    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p" event={"ID":"c34fc4a6-58be-48e7-83db-1a0d5619031e","Type":"ContainerStarted","Data":"9eaec0adcc5d7b8a5c0cbef3ecb95aceb36849ceaa4a6d3134516baa742b8a42"}
Jul 02 12:39:35.617712 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:35.617657    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p" event={"ID":"c34fc4a6-58be-48e7-83db-1a0d5619031e","Type":"ContainerStarted","Data":"271df127622c03d02e29dee01f105e1ac1b608a8c8a65f58056219f5e25e2ef0"}
Jul 02 12:39:35.617712 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:35.617688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p" event={"ID":"c34fc4a6-58be-48e7-83db-1a0d5619031e","Type":"ContainerStarted","Data":"1a22f1300dc306b8c1c8eaba431f19055d4d8ff089218e391c17b41994e102f4"}
Jul 02 12:39:35.617778 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:35.617715    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p" event={"ID":"c34fc4a6-58be-48e7-83db-1a0d5619031e","Type":"ContainerStarted","Data":"040f923cdc31c33f45cb9b0f8822b8c85eeff0f36ae4588ad245bea8bf32902a"}
Jul 02 12:39:35.617778 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:35.617742    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p" event={"ID":"c34fc4a6-58be-48e7-83db-1a0d5619031e","Type":"ContainerStarted","Data":"099ac386f9cb6a71a4a9ae0f8da0abc5783af6d8c5d57311b35acaae8eeaf878"}
Jul 02 12:39:35.640941 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:35.640902    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2bmr8" podStartSLOduration=4.861794328 podStartE2EDuration="23.640891403s" totalImagesPullingTime="18.779097075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:35.640391931 +0000 UTC m=+23.623865619" watchObservedRunningTime="2026-07-02 12:39:35.640891403 +0000 UTC m=+23.624365086"
Jul 02 12:39:35.734916 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:35.734889    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 12:39:36.417878 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:36.417843    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:36.418207 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:36.417857    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:36.418207 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:36.417965    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:36.418207 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:36.418087    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:36.418207 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:36.418143    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:36.418412 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:36.418206    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:36.436034 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:36.435932    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T12:39:35.734904823Z","UUID":"4c68ce9e-0f64-402a-971a-c3ef186b852b","Handler":null,"Name":"","Endpoint":""}
Jul 02 12:39:36.437595 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:36.437565    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 12:39:36.437709 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:36.437601    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 12:39:36.621545 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:36.621511    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57" event={"ID":"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c","Type":"ContainerStarted","Data":"54103c72c3d1b2d33a302baa735b3e24bc743d2bb7ff14cd7dc75bce7d159f67"}
Jul 02 12:39:37.626936 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:37.626628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57" event={"ID":"ddb014eb-46fe-43cf-bcdc-e7dcfc07993c","Type":"ContainerStarted","Data":"b2059e88d23cf3c4cf1b35a3fcd0d9491366bffd4035c273ae3223a62013ad4d"}
Jul 02 12:39:37.630437 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:37.630404    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p" event={"ID":"c34fc4a6-58be-48e7-83db-1a0d5619031e","Type":"ContainerStarted","Data":"8c9d7be49ceaac2734105a0f4e130a8e5a53c07faabfb1ba8b30d36100d67709"}
Jul 02 12:39:38.417265 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:38.417227    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:38.417265 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:38.417264    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:38.417569 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:38.417229    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:38.417569 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:38.417389    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:38.417569 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:38.417490    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:38.417726 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:38.417595    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:39.525375 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:39.525342    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nmzpg"
Jul 02 12:39:39.526191 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:39.526168    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nmzpg"
Jul 02 12:39:39.548876 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:39.548790    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8z57" podStartSLOduration=6.6786357 podStartE2EDuration="27.548758556s" totalImagesPullingTime="20.870122856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:37.652663431 +0000 UTC m=+25.636137115" watchObservedRunningTime="2026-07-02 12:39:39.548758556 +0000 UTC m=+27.532232236"
Jul 02 12:39:39.635030 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:39.635005    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nmzpg"
Jul 02 12:39:39.635564 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:39.635547    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nmzpg"
Jul 02 12:39:40.417797 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:40.417756    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:40.417961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:40.417757    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:40.417961 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:40.417878    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:40.417961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:40.417896    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:40.418127 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:40.417965    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:40.418127 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:40.418033    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:40.640647 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:40.640613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p" event={"ID":"c34fc4a6-58be-48e7-83db-1a0d5619031e","Type":"ContainerStarted","Data":"317528a3142ebbc573cfe910f855732e570f74a68fe8e25858ed64c669d0b0ec"}
Jul 02 12:39:40.641479 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:40.640911    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:40.642514 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:40.642493    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40" containerID="3adf72aa8e83746e6f88e70483af1b0359e2ddbcf836b4435782b820de61be24" exitCode=0
Jul 02 12:39:40.642629 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:40.642576    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2cwd" event={"ID":"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40","Type":"ContainerDied","Data":"3adf72aa8e83746e6f88e70483af1b0359e2ddbcf836b4435782b820de61be24"}
Jul 02 12:39:40.656381 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:40.656361    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:40.677783 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:40.677717    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p" podStartSLOduration=9.841171466 podStartE2EDuration="28.677703708s" totalImagesPullingTime="18.836532242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:40.677370981 +0000 UTC m=+28.660844664" watchObservedRunningTime="2026-07-02 12:39:40.677703708 +0000 UTC m=+28.661177433"
Jul 02 12:39:41.647821 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:41.647798    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:41.648280 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:41.647886    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:41.664678 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:41.664655    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:39:41.674185 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:41.674163    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v7gwd"]
Jul 02 12:39:41.674290 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:41.674253    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:41.674377 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:41.674358    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:41.677468 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:41.677250    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bx9gl"]
Jul 02 12:39:41.677468 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:41.677358    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:41.677468 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:41.677439    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:41.677898 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:41.677878    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r975p"]
Jul 02 12:39:41.677966 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:41.677955    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:41.678171 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:41.678022    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:42.651577 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:42.651547    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40" containerID="22c5839b9bd7f244653127199012763493188440cc1310767ecadab59006a67d" exitCode=0
Jul 02 12:39:42.652077 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:42.651638    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2cwd" event={"ID":"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40","Type":"ContainerDied","Data":"22c5839b9bd7f244653127199012763493188440cc1310767ecadab59006a67d"}
Jul 02 12:39:43.417768 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:43.417609    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:43.417883 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:43.417662    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:43.417883 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:43.417841    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:43.417998 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:43.417662    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:43.417998 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:43.417974    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:43.418098 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:43.418066    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:44.657821 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:44.657788    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40" containerID="1fd5b26afbb4f12caae19489ebc911ccf8bb97426520bf20f1083ce7e3086b11" exitCode=0
Jul 02 12:39:44.658385 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:44.657844    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2cwd" event={"ID":"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40","Type":"ContainerDied","Data":"1fd5b26afbb4f12caae19489ebc911ccf8bb97426520bf20f1083ce7e3086b11"}
Jul 02 12:39:45.416820 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:45.416781    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:45.416820 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:45.416810    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:45.417058 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:45.416810    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:45.417058 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:45.416904    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx9gl" podUID="1da27b61-6566-4583-a851-61ea1f1e8d11"
Jul 02 12:39:45.417058 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:45.417007    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v7gwd" podUID="dc387565-3fa1-4271-a12e-dda0c699e3e4"
Jul 02 12:39:45.417178 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:45.417101    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r975p" podUID="2697b9b4-7477-4e35-a9f3-298ed7a423ed"
Jul 02 12:39:46.116168 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:46.116135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:46.116725 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:46.116321    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:46.116725 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:46.116381    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs podName:dc387565-3fa1-4271-a12e-dda0c699e3e4 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:18.116367277 +0000 UTC m=+66.099840943 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs") pod "network-metrics-daemon-v7gwd" (UID: "dc387565-3fa1-4271-a12e-dda0c699e3e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:46.216670 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:46.216622    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zck\" (UniqueName: \"kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck\") pod \"network-check-target-r975p\" (UID: \"2697b9b4-7477-4e35-a9f3-298ed7a423ed\") " pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:46.216854 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:46.216689    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:46.216854 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:46.216807    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:46.216854 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:46.216835    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:39:46.216854 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:46.216856    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:39:46.217003 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:46.216867    2564 projected.go:196] Error preparing data for projected volume kube-api-access-k2zck for pod openshift-network-diagnostics/network-check-target-r975p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:46.217003 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:46.216872    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret podName:1da27b61-6566-4583-a851-61ea1f1e8d11 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:18.216854527 +0000 UTC m=+66.200328193 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret") pod "global-pull-secret-syncer-bx9gl" (UID: "1da27b61-6566-4583-a851-61ea1f1e8d11") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:39:46.217003 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:46.216917    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck podName:2697b9b4-7477-4e35-a9f3-298ed7a423ed nodeName:}" failed. No retries permitted until 2026-07-02 12:40:18.21689976 +0000 UTC m=+66.200373483 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2zck" (UniqueName: "kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck") pod "network-check-target-r975p" (UID: "2697b9b4-7477-4e35-a9f3-298ed7a423ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:47.257610 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.257542    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeReady"
Jul 02 12:39:47.257991 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.257671    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 12:39:47.350123 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.350093    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-df765d5c7-sj6rs"]
Jul 02 12:39:47.352291 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.352269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.364046 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.364014    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 12:39:47.364169 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.364140    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 12:39:47.364228 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.364179    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 12:39:47.365275 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.365254    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s268d\""
Jul 02 12:39:47.384626 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.384599    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 12:39:47.385099 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.385075    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-df765d5c7-sj6rs"]
Jul 02 12:39:47.402519 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.402489    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-drvvs"]
Jul 02 12:39:47.404313 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.404283    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-drvvs"
Jul 02 12:39:47.408579 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.408563    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 12:39:47.408718 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.408691    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 12:39:47.411331 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.411238    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-78pzs\""
Jul 02 12:39:47.414587 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.414569    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 12:39:47.416753 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.416729    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:39:47.416862 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.416761    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:39:47.416908 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.416894    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:39:47.421166 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.421128    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 12:39:47.421281 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.421225    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nktrd\""
Jul 02 12:39:47.421491 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.421476    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 12:39:47.422373 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.422337    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 12:39:47.422975 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.422577    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 12:39:47.422975 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.422597    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fnc5f\""
Jul 02 12:39:47.423477 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.423455    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5bz8v"]
Jul 02 12:39:47.426602 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.426584    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-drvvs"]
Jul 02 12:39:47.426702 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.426678    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:47.434814 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.434794    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 12:39:47.434907 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.434873    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5bsnw\""
Jul 02 12:39:47.434989 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.434795    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 12:39:47.457189 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.457155    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5bz8v"]
Jul 02 12:39:47.525874 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.525802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5djp5\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-kube-api-access-5djp5\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.525874 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.525832    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7c2j\" (UniqueName: \"kubernetes.io/projected/0318470e-cccb-4d26-96b4-3b464de83f56-kube-api-access-g7c2j\") pod \"ingress-canary-drvvs\" (UID: \"0318470e-cccb-4d26-96b4-3b464de83f56\") " pod="openshift-ingress-canary/ingress-canary-drvvs"
Jul 02 12:39:47.525874 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.525856    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-bound-sa-token\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.525874 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.525876    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:47.526142 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.525943    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n9kx\" (UniqueName: \"kubernetes.io/projected/d173e052-416f-46c8-a7ed-b3b779c66c66-kube-api-access-6n9kx\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:47.526142 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.525990    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bc1bee4-4948-4f8b-905a-875187f12659-installation-pull-secrets\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.526142 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.526025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bc1bee4-4948-4f8b-905a-875187f12659-trusted-ca\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.526142 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.526058    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bc1bee4-4948-4f8b-905a-875187f12659-image-registry-private-configuration\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.526142 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.526088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bc1bee4-4948-4f8b-905a-875187f12659-registry-certificates\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.526356 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.526172    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bc1bee4-4948-4f8b-905a-875187f12659-ca-trust-extracted\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.526356 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.526206    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d173e052-416f-46c8-a7ed-b3b779c66c66-tmp-dir\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:47.526356 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.526228    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.526356 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.526260    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d173e052-416f-46c8-a7ed-b3b779c66c66-config-volume\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:47.526356 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.526286    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0318470e-cccb-4d26-96b4-3b464de83f56-cert\") pod \"ingress-canary-drvvs\" (UID: \"0318470e-cccb-4d26-96b4-3b464de83f56\") " pod="openshift-ingress-canary/ingress-canary-drvvs"
Jul 02 12:39:47.627586 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.627552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d173e052-416f-46c8-a7ed-b3b779c66c66-config-volume\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:47.627770 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.627702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0318470e-cccb-4d26-96b4-3b464de83f56-cert\") pod \"ingress-canary-drvvs\" (UID: \"0318470e-cccb-4d26-96b4-3b464de83f56\") " pod="openshift-ingress-canary/ingress-canary-drvvs"
Jul 02 12:39:47.627838 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.627787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5djp5\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-kube-api-access-5djp5\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.627838 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.627820    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7c2j\" (UniqueName: \"kubernetes.io/projected/0318470e-cccb-4d26-96b4-3b464de83f56-kube-api-access-g7c2j\") pod \"ingress-canary-drvvs\" (UID: \"0318470e-cccb-4d26-96b4-3b464de83f56\") " pod="openshift-ingress-canary/ingress-canary-drvvs"
Jul 02 12:39:47.627938 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.627849    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-bound-sa-token\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.627938 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.627877    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:47.627938 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.627903    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n9kx\" (UniqueName: \"kubernetes.io/projected/d173e052-416f-46c8-a7ed-b3b779c66c66-kube-api-access-6n9kx\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:47.627938 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.627926    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bc1bee4-4948-4f8b-905a-875187f12659-installation-pull-secrets\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.628125 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.627956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bc1bee4-4948-4f8b-905a-875187f12659-trusted-ca\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.628125 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.627986    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bc1bee4-4948-4f8b-905a-875187f12659-image-registry-private-configuration\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.628125 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.628017    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bc1bee4-4948-4f8b-905a-875187f12659-registry-certificates\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.628125 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.628077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bc1bee4-4948-4f8b-905a-875187f12659-ca-trust-extracted\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.628125 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.628113    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d173e052-416f-46c8-a7ed-b3b779c66c66-tmp-dir\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:47.628395 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.628136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.628395 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:47.628254    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 12:39:47.628395 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:47.628268    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-df765d5c7-sj6rs: secret "image-registry-tls" not found
Jul 02 12:39:47.628395 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.628270    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d173e052-416f-46c8-a7ed-b3b779c66c66-config-volume\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:47.628395 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:47.628347    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls podName:3bc1bee4-4948-4f8b-905a-875187f12659 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:48.128330247 +0000 UTC m=+36.111803925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls") pod "image-registry-df765d5c7-sj6rs" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659") : secret "image-registry-tls" not found
Jul 02 12:39:47.628639 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:47.628399    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:47.628639 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:47.628467    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls podName:d173e052-416f-46c8-a7ed-b3b779c66c66 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:48.128442788 +0000 UTC m=+36.111916468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls") pod "dns-default-5bz8v" (UID: "d173e052-416f-46c8-a7ed-b3b779c66c66") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:47.628745 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.628660    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d173e052-416f-46c8-a7ed-b3b779c66c66-tmp-dir\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:47.628870 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.628830    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bc1bee4-4948-4f8b-905a-875187f12659-ca-trust-extracted\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.628984 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.628959    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bc1bee4-4948-4f8b-905a-875187f12659-registry-certificates\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.629085 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.629060    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bc1bee4-4948-4f8b-905a-875187f12659-trusted-ca\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.632625 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.632463    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bc1bee4-4948-4f8b-905a-875187f12659-installation-pull-secrets\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.632732 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.632538    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bc1bee4-4948-4f8b-905a-875187f12659-image-registry-private-configuration\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.632732 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.632546    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0318470e-cccb-4d26-96b4-3b464de83f56-cert\") pod \"ingress-canary-drvvs\" (UID: \"0318470e-cccb-4d26-96b4-3b464de83f56\") " pod="openshift-ingress-canary/ingress-canary-drvvs"
Jul 02 12:39:47.647642 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.647623    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7c2j\" (UniqueName: \"kubernetes.io/projected/0318470e-cccb-4d26-96b4-3b464de83f56-kube-api-access-g7c2j\") pod \"ingress-canary-drvvs\" (UID: \"0318470e-cccb-4d26-96b4-3b464de83f56\") " pod="openshift-ingress-canary/ingress-canary-drvvs"
Jul 02 12:39:47.647827 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.647808    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n9kx\" (UniqueName: \"kubernetes.io/projected/d173e052-416f-46c8-a7ed-b3b779c66c66-kube-api-access-6n9kx\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:47.655818 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.655795    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-bound-sa-token\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.656215 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.656192    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5djp5\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-kube-api-access-5djp5\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:47.714417 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:47.714382    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-drvvs"
Jul 02 12:39:48.131476 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:48.131436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:48.131690 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:48.131535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:48.131690 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:48.131597    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 12:39:48.131690 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:48.131619    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-df765d5c7-sj6rs: secret "image-registry-tls" not found
Jul 02 12:39:48.131690 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:48.131652    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:48.131690 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:48.131674    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls podName:3bc1bee4-4948-4f8b-905a-875187f12659 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:49.131659974 +0000 UTC m=+37.115133635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls") pod "image-registry-df765d5c7-sj6rs" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659") : secret "image-registry-tls" not found
Jul 02 12:39:48.131899 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:48.131713    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls podName:d173e052-416f-46c8-a7ed-b3b779c66c66 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:49.131697067 +0000 UTC m=+37.115170749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls") pod "dns-default-5bz8v" (UID: "d173e052-416f-46c8-a7ed-b3b779c66c66") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:48.841279 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:48.841256    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-drvvs"]
Jul 02 12:39:48.845227 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:48.845206    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0318470e_cccb_4d26_96b4_3b464de83f56.slice/crio-d11f4576d4410d36853ffdc03499a5dba586d3f18e4823f55ad51ae25de30194 WatchSource:0}: Error finding container d11f4576d4410d36853ffdc03499a5dba586d3f18e4823f55ad51ae25de30194: Status 404 returned error can't find the container with id d11f4576d4410d36853ffdc03499a5dba586d3f18e4823f55ad51ae25de30194
Jul 02 12:39:49.139638 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:49.139563    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:49.139638 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:49.139626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:49.139807 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:49.139707    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 12:39:49.139807 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:49.139727    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-df765d5c7-sj6rs: secret "image-registry-tls" not found
Jul 02 12:39:49.139807 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:49.139729    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:49.139807 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:49.139786    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls podName:d173e052-416f-46c8-a7ed-b3b779c66c66 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:51.139773511 +0000 UTC m=+39.123247172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls") pod "dns-default-5bz8v" (UID: "d173e052-416f-46c8-a7ed-b3b779c66c66") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:49.139807 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:49.139800    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls podName:3bc1bee4-4948-4f8b-905a-875187f12659 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:51.13979491 +0000 UTC m=+39.123268571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls") pod "image-registry-df765d5c7-sj6rs" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659") : secret "image-registry-tls" not found
Jul 02 12:39:49.674458 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:49.674424    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40" containerID="8cf5d7a468ae9795af1d59753d40c855791604203ce1f02db4a2b622f19ad764" exitCode=0
Jul 02 12:39:49.674628 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:49.674507    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2cwd" event={"ID":"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40","Type":"ContainerDied","Data":"8cf5d7a468ae9795af1d59753d40c855791604203ce1f02db4a2b622f19ad764"}
Jul 02 12:39:49.675797 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:49.675775    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-drvvs" event={"ID":"0318470e-cccb-4d26-96b4-3b464de83f56","Type":"ContainerStarted","Data":"d11f4576d4410d36853ffdc03499a5dba586d3f18e4823f55ad51ae25de30194"}
Jul 02 12:39:50.682501 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:50.682464    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40" containerID="9176273f96731d5a29105f7f8ae233b5968596c0b6d2c303caa60033ba8de2bb" exitCode=0
Jul 02 12:39:50.682906 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:50.682524    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2cwd" event={"ID":"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40","Type":"ContainerDied","Data":"9176273f96731d5a29105f7f8ae233b5968596c0b6d2c303caa60033ba8de2bb"}
Jul 02 12:39:51.156445 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:51.156283    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:51.156550 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:51.156487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:51.156550 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:51.156417    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 12:39:51.156550 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:51.156530    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-df765d5c7-sj6rs: secret "image-registry-tls" not found
Jul 02 12:39:51.156685 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:51.156580    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:51.156685 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:51.156594    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls podName:3bc1bee4-4948-4f8b-905a-875187f12659 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:55.156572015 +0000 UTC m=+43.140045698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls") pod "image-registry-df765d5c7-sj6rs" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659") : secret "image-registry-tls" not found
Jul 02 12:39:51.156685 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:51.156618    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls podName:d173e052-416f-46c8-a7ed-b3b779c66c66 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:55.156605797 +0000 UTC m=+43.140079458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls") pod "dns-default-5bz8v" (UID: "d173e052-416f-46c8-a7ed-b3b779c66c66") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:51.688490 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:51.688461    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2cwd" event={"ID":"cb3c40cd-00cb-4a1b-82a8-dfde87d1fd40","Type":"ContainerStarted","Data":"6d417857aadff9d19b875889b86014d5dd56ad4e91a5b564c07c18d1ae986feb"}
Jul 02 12:39:51.690139 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:51.690107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-drvvs" event={"ID":"0318470e-cccb-4d26-96b4-3b464de83f56","Type":"ContainerStarted","Data":"857571688de776c50fabebeb5922169866cc590b12a1fdfd8d88a271ab3bb8ab"}
Jul 02 12:39:51.740403 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:51.740353    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-drvvs" podStartSLOduration=2.178979217 podStartE2EDuration="4.740337682s" totalImagesPullingTime="2.561358465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:51.739263964 +0000 UTC m=+39.722737660" watchObservedRunningTime="2026-07-02 12:39:51.740337682 +0000 UTC m=+39.723811364"
Jul 02 12:39:51.740605 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:51.740581    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j2cwd" podStartSLOduration=10.556587142 podStartE2EDuration="39.740577612s" totalImagesPullingTime="29.18399047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:51.720025552 +0000 UTC m=+39.703499238" watchObservedRunningTime="2026-07-02 12:39:51.740577612 +0000 UTC m=+39.724051295"
Jul 02 12:39:55.184732 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:55.184685    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:39:55.185168 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:55.184755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:39:55.185168 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:55.184837    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 12:39:55.185168 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:55.184847    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-df765d5c7-sj6rs: secret "image-registry-tls" not found
Jul 02 12:39:55.185168 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:55.184844    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:55.185168 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:55.184927    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls podName:3bc1bee4-4948-4f8b-905a-875187f12659 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:03.184883201 +0000 UTC m=+51.168356867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls") pod "image-registry-df765d5c7-sj6rs" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659") : secret "image-registry-tls" not found
Jul 02 12:39:55.185168 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:55.184940    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls podName:d173e052-416f-46c8-a7ed-b3b779c66c66 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:03.184934007 +0000 UTC m=+51.168407668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls") pod "dns-default-5bz8v" (UID: "d173e052-416f-46c8-a7ed-b3b779c66c66") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:57.408480 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.408447    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5f67b55555-v24km"]
Jul 02 12:39:57.435883 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.435858    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-v24km"]
Jul 02 12:39:57.436008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.435940    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.438715 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.438695    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 02 12:39:57.438844 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.438739    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 02 12:39:57.439082 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.439067    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 12:39:57.439789 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.439773    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-kztgz\""
Jul 02 12:39:57.439911 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.439892    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 12:39:57.444861 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.444845    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 02 12:39:57.498336 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.498313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.498425 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.498354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-service-ca-bundle\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.498425 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.498383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-tmp\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.498425 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.498400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-snapshots\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.498526 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.498443    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-serving-cert\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.498526 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.498459    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr82v\" (UniqueName: \"kubernetes.io/projected/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-kube-api-access-nr82v\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.512336 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.512312    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"]
Jul 02 12:39:57.526022 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.525996    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"]
Jul 02 12:39:57.526134 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.526106    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:39:57.529195 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.529168    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 02 12:39:57.529329 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.529176    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 12:39:57.531035 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.530905    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 12:39:57.531035 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.530922    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 02 12:39:57.531035 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.530930    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8xhgv\""
Jul 02 12:39:57.542347 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.542329    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"]
Jul 02 12:39:57.542445 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.542363    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"]
Jul 02 12:39:57.542486 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.542443    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"
Jul 02 12:39:57.545137 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.545119    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 02 12:39:57.545137 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.545127    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 02 12:39:57.545286 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.545187    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 02 12:39:57.545482 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.545467    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 02 12:39:57.545816 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.545803    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-25fm4\""
Jul 02 12:39:57.599291 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-tmp\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.599291 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-snapshots\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.599479 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599331    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hgvx\" (UniqueName: \"kubernetes.io/projected/33aa4b59-fdea-4a0e-8b70-76b57da84a95-kube-api-access-4hgvx\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:39:57.599479 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599370    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a15051df-16fa-46c6-8271-9351e5c7e12c-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-kjpgf\" (UID: \"a15051df-16fa-46c6-8271-9351e5c7e12c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"
Jul 02 12:39:57.599479 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599396    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-serving-cert\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.599479 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr82v\" (UniqueName: \"kubernetes.io/projected/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-kube-api-access-nr82v\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.599479 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599426    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xshv\" (UniqueName: \"kubernetes.io/projected/a15051df-16fa-46c6-8271-9351e5c7e12c-kube-api-access-7xshv\") pod \"service-ca-operator-5d7b7dc95c-kjpgf\" (UID: \"a15051df-16fa-46c6-8271-9351e5c7e12c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"
Jul 02 12:39:57.599681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599623    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/33aa4b59-fdea-4a0e-8b70-76b57da84a95-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:39:57.599681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599640    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-tmp\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.599681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599662    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a15051df-16fa-46c6-8271-9351e5c7e12c-config\") pod \"service-ca-operator-5d7b7dc95c-kjpgf\" (UID: \"a15051df-16fa-46c6-8271-9351e5c7e12c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"
Jul 02 12:39:57.599791 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.599791 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:39:57.599896 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-service-ca-bundle\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.599994 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.599963    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-snapshots\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.600245 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.600230    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-service-ca-bundle\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.600588 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.600572    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.603464 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.603446    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-serving-cert\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.613247 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.613224    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr82v\" (UniqueName: \"kubernetes.io/projected/49a6bea0-1155-4bd8-b8f6-f6b5a5016167-kube-api-access-nr82v\") pod \"insights-operator-5f67b55555-v24km\" (UID: \"49a6bea0-1155-4bd8-b8f6-f6b5a5016167\") " pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.700584 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.700558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/33aa4b59-fdea-4a0e-8b70-76b57da84a95-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:39:57.700719 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.700588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a15051df-16fa-46c6-8271-9351e5c7e12c-config\") pod \"service-ca-operator-5d7b7dc95c-kjpgf\" (UID: \"a15051df-16fa-46c6-8271-9351e5c7e12c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"
Jul 02 12:39:57.700719 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.700608    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:39:57.700719 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.700641    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hgvx\" (UniqueName: \"kubernetes.io/projected/33aa4b59-fdea-4a0e-8b70-76b57da84a95-kube-api-access-4hgvx\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:39:57.700719 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.700714    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a15051df-16fa-46c6-8271-9351e5c7e12c-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-kjpgf\" (UID: \"a15051df-16fa-46c6-8271-9351e5c7e12c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"
Jul 02 12:39:57.700922 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.700760    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xshv\" (UniqueName: \"kubernetes.io/projected/a15051df-16fa-46c6-8271-9351e5c7e12c-kube-api-access-7xshv\") pod \"service-ca-operator-5d7b7dc95c-kjpgf\" (UID: \"a15051df-16fa-46c6-8271-9351e5c7e12c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"
Jul 02 12:39:57.700922 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:57.700885    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:57.701019 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:57.700958    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls podName:33aa4b59-fdea-4a0e-8b70-76b57da84a95 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:58.200932857 +0000 UTC m=+46.184406525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-m76wm" (UID: "33aa4b59-fdea-4a0e-8b70-76b57da84a95") : secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:57.701218 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.701196    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a15051df-16fa-46c6-8271-9351e5c7e12c-config\") pod \"service-ca-operator-5d7b7dc95c-kjpgf\" (UID: \"a15051df-16fa-46c6-8271-9351e5c7e12c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"
Jul 02 12:39:57.701285 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.701256    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/33aa4b59-fdea-4a0e-8b70-76b57da84a95-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:39:57.702881 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.702859    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a15051df-16fa-46c6-8271-9351e5c7e12c-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-kjpgf\" (UID: \"a15051df-16fa-46c6-8271-9351e5c7e12c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"
Jul 02 12:39:57.708962 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.708943    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xshv\" (UniqueName: \"kubernetes.io/projected/a15051df-16fa-46c6-8271-9351e5c7e12c-kube-api-access-7xshv\") pod \"service-ca-operator-5d7b7dc95c-kjpgf\" (UID: \"a15051df-16fa-46c6-8271-9351e5c7e12c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"
Jul 02 12:39:57.713278 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.713258    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hgvx\" (UniqueName: \"kubernetes.io/projected/33aa4b59-fdea-4a0e-8b70-76b57da84a95-kube-api-access-4hgvx\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:39:57.745073 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.745052    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-v24km"
Jul 02 12:39:57.850578 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.850555    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"
Jul 02 12:39:57.887420 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:57.887392    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-v24km"]
Jul 02 12:39:57.890255 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:57.890227    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a6bea0_1155_4bd8_b8f6_f6b5a5016167.slice/crio-0c76d6b880047296c82b5dc00b2f28765c963d424beedcd4479423b5bff49578 WatchSource:0}: Error finding container 0c76d6b880047296c82b5dc00b2f28765c963d424beedcd4479423b5bff49578: Status 404 returned error can't find the container with id 0c76d6b880047296c82b5dc00b2f28765c963d424beedcd4479423b5bff49578
Jul 02 12:39:58.011660 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:58.011635    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf"]
Jul 02 12:39:58.014541 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:39:58.014514    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15051df_16fa_46c6_8271_9351e5c7e12c.slice/crio-30947efd94cd9fdcae633b3305800472460b48d394d3381da1f288f9b31d04b9 WatchSource:0}: Error finding container 30947efd94cd9fdcae633b3305800472460b48d394d3381da1f288f9b31d04b9: Status 404 returned error can't find the container with id 30947efd94cd9fdcae633b3305800472460b48d394d3381da1f288f9b31d04b9
Jul 02 12:39:58.205104 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:58.205069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:39:58.205244 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:58.205193    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:58.205244 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:58.205244    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls podName:33aa4b59-fdea-4a0e-8b70-76b57da84a95 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:59.2052313 +0000 UTC m=+47.188704961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-m76wm" (UID: "33aa4b59-fdea-4a0e-8b70-76b57da84a95") : secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:58.707790 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:58.707751    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf" event={"ID":"a15051df-16fa-46c6-8271-9351e5c7e12c","Type":"ContainerStarted","Data":"30947efd94cd9fdcae633b3305800472460b48d394d3381da1f288f9b31d04b9"}
Jul 02 12:39:58.708980 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:58.708953    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-v24km" event={"ID":"49a6bea0-1155-4bd8-b8f6-f6b5a5016167","Type":"ContainerStarted","Data":"0c76d6b880047296c82b5dc00b2f28765c963d424beedcd4479423b5bff49578"}
Jul 02 12:39:59.213131 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:39:59.213091    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:39:59.213339 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:59.213284    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:59.213402 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:39:59.213382    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls podName:33aa4b59-fdea-4a0e-8b70-76b57da84a95 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:01.213360736 +0000 UTC m=+49.196834406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-m76wm" (UID: "33aa4b59-fdea-4a0e-8b70-76b57da84a95") : secret "cluster-monitoring-operator-tls" not found
Jul 02 12:40:01.228521 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:01.228478    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:40:01.228888 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:01.228632    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 12:40:01.228888 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:01.228701    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls podName:33aa4b59-fdea-4a0e-8b70-76b57da84a95 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:05.228681802 +0000 UTC m=+53.212155464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-m76wm" (UID: "33aa4b59-fdea-4a0e-8b70-76b57da84a95") : secret "cluster-monitoring-operator-tls" not found
Jul 02 12:40:01.718317 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:01.718263    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf" event={"ID":"a15051df-16fa-46c6-8271-9351e5c7e12c","Type":"ContainerStarted","Data":"d9532706871e5885bddf39aa9b4faef96226476d1ac063367e740e12027400a7"}
Jul 02 12:40:01.719964 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:01.719929    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-v24km" event={"ID":"49a6bea0-1155-4bd8-b8f6-f6b5a5016167","Type":"ContainerStarted","Data":"27d9a7424740a3ba09a511076c4dffad0c1e26d14865a90b39c9902044a3c746"}
Jul 02 12:40:01.735269 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:01.735220    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf" podStartSLOduration=1.718640419 podStartE2EDuration="4.735201077s" totalImagesPullingTime="3.016560658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:01.734948264 +0000 UTC m=+49.718421945" watchObservedRunningTime="2026-07-02 12:40:01.735201077 +0000 UTC m=+49.718674765"
Jul 02 12:40:01.755851 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:01.755798    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5f67b55555-v24km" podStartSLOduration=1.62080569 podStartE2EDuration="4.755778222s" totalImagesPullingTime="3.134972532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:01.75549293 +0000 UTC m=+49.738966613" watchObservedRunningTime="2026-07-02 12:40:01.755778222 +0000 UTC m=+49.739251886"
Jul 02 12:40:02.035204 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.035123    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ksn6z"]
Jul 02 12:40:02.061620 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.061585    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ksn6z"]
Jul 02 12:40:02.061771 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.061723    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.064638 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.064612    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 12:40:02.064883 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.064857    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 12:40:02.065013 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.064857    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-njqpt\""
Jul 02 12:40:02.135356 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.135322    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.135494 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.135426    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf13802d-96c8-49d1-b303-ee39802c83f3-crio-socket\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.135494 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.135474    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf13802d-96c8-49d1-b303-ee39802c83f3-data-volume\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.135580 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.135519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf13802d-96c8-49d1-b303-ee39802c83f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.135580 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.135534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7rd5\" (UniqueName: \"kubernetes.io/projected/bf13802d-96c8-49d1-b303-ee39802c83f3-kube-api-access-b7rd5\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.236632 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.236591    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf13802d-96c8-49d1-b303-ee39802c83f3-crio-socket\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.237094 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.236647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf13802d-96c8-49d1-b303-ee39802c83f3-data-volume\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.237094 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.236799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf13802d-96c8-49d1-b303-ee39802c83f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.237094 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.236830    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7rd5\" (UniqueName: \"kubernetes.io/projected/bf13802d-96c8-49d1-b303-ee39802c83f3-kube-api-access-b7rd5\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.237094 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.236857    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf13802d-96c8-49d1-b303-ee39802c83f3-crio-socket\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.237094 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:02.236915    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 12:40:02.237094 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.236927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.237094 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.236917    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf13802d-96c8-49d1-b303-ee39802c83f3-data-volume\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.237094 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:02.236997    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf13802d-96c8-49d1-b303-ee39802c83f3-kube-rbac-proxy-cm podName:bf13802d-96c8-49d1-b303-ee39802c83f3 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:02.736976966 +0000 UTC m=+50.720450631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf13802d-96c8-49d1-b303-ee39802c83f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ksn6z" (UID: "bf13802d-96c8-49d1-b303-ee39802c83f3") : configmap "kube-rbac-proxy" not found
Jul 02 12:40:02.237094 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:02.237050    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:40:02.237094 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:02.237096    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls podName:bf13802d-96c8-49d1-b303-ee39802c83f3 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:02.737081164 +0000 UTC m=+50.720554864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ksn6z" (UID: "bf13802d-96c8-49d1-b303-ee39802c83f3") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:40:02.246059 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.246037    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7rd5\" (UniqueName: \"kubernetes.io/projected/bf13802d-96c8-49d1-b303-ee39802c83f3-kube-api-access-b7rd5\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.741371 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.741341    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.741541 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:02.741470    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:40:02.741541 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:02.741508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf13802d-96c8-49d1-b303-ee39802c83f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:02.741541 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:02.741530    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls podName:bf13802d-96c8-49d1-b303-ee39802c83f3 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:03.741513425 +0000 UTC m=+51.724987092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ksn6z" (UID: "bf13802d-96c8-49d1-b303-ee39802c83f3") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:40:02.741733 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:02.741558    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 12:40:02.741733 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:02.741617    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf13802d-96c8-49d1-b303-ee39802c83f3-kube-rbac-proxy-cm podName:bf13802d-96c8-49d1-b303-ee39802c83f3 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:03.741595059 +0000 UTC m=+51.725068733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf13802d-96c8-49d1-b303-ee39802c83f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ksn6z" (UID: "bf13802d-96c8-49d1-b303-ee39802c83f3") : configmap "kube-rbac-proxy" not found
Jul 02 12:40:03.245531 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:03.245491    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:40:03.245899 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:03.245555    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:40:03.245899 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:03.245653    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:40:03.245899 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:03.245698    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 12:40:03.245899 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:03.245710    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-df765d5c7-sj6rs: secret "image-registry-tls" not found
Jul 02 12:40:03.245899 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:03.245720    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls podName:d173e052-416f-46c8-a7ed-b3b779c66c66 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:19.245702262 +0000 UTC m=+67.229175923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls") pod "dns-default-5bz8v" (UID: "d173e052-416f-46c8-a7ed-b3b779c66c66") : secret "dns-default-metrics-tls" not found
Jul 02 12:40:03.245899 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:03.245758    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls podName:3bc1bee4-4948-4f8b-905a-875187f12659 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:19.245747079 +0000 UTC m=+67.229220744 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls") pod "image-registry-df765d5c7-sj6rs" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659") : secret "image-registry-tls" not found
Jul 02 12:40:03.749499 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:03.749469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:03.749671 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:03.749622    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:40:03.749740 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:03.749629    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf13802d-96c8-49d1-b303-ee39802c83f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:03.749740 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:03.749682    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls podName:bf13802d-96c8-49d1-b303-ee39802c83f3 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:05.749665684 +0000 UTC m=+53.733139345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ksn6z" (UID: "bf13802d-96c8-49d1-b303-ee39802c83f3") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:40:03.750043 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:03.750027    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf13802d-96c8-49d1-b303-ee39802c83f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:04.229693 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.229665    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lczx4_e6dde057-4214-4246-bea1-a0820ab52a59/dns-node-resolver/0.log"
Jul 02 12:40:04.445362 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.445332    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"]
Jul 02 12:40:04.458092 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.458065    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"]
Jul 02 12:40:04.458219 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.458153    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"
Jul 02 12:40:04.461196 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.461169    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-v4qbf\""
Jul 02 12:40:04.461398 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.461381    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 02 12:40:04.461467 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.461447    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 02 12:40:04.462388 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.462372    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 02 12:40:04.462567 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.462554    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 02 12:40:04.557045 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.556969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/25db85a8-4af1-4c43-b29d-4e65e736d780-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-twwbr\" (UID: \"25db85a8-4af1-4c43-b29d-4e65e736d780\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"
Jul 02 12:40:04.557045 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.557018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76pp2\" (UniqueName: \"kubernetes.io/projected/25db85a8-4af1-4c43-b29d-4e65e736d780-kube-api-access-76pp2\") pod \"service-ca-7bb4f4bcc7-twwbr\" (UID: \"25db85a8-4af1-4c43-b29d-4e65e736d780\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"
Jul 02 12:40:04.557229 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.557156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/25db85a8-4af1-4c43-b29d-4e65e736d780-signing-key\") pod \"service-ca-7bb4f4bcc7-twwbr\" (UID: \"25db85a8-4af1-4c43-b29d-4e65e736d780\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"
Jul 02 12:40:04.657989 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.657957    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/25db85a8-4af1-4c43-b29d-4e65e736d780-signing-key\") pod \"service-ca-7bb4f4bcc7-twwbr\" (UID: \"25db85a8-4af1-4c43-b29d-4e65e736d780\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"
Jul 02 12:40:04.657989 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.657993    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/25db85a8-4af1-4c43-b29d-4e65e736d780-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-twwbr\" (UID: \"25db85a8-4af1-4c43-b29d-4e65e736d780\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"
Jul 02 12:40:04.658202 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.658027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76pp2\" (UniqueName: \"kubernetes.io/projected/25db85a8-4af1-4c43-b29d-4e65e736d780-kube-api-access-76pp2\") pod \"service-ca-7bb4f4bcc7-twwbr\" (UID: \"25db85a8-4af1-4c43-b29d-4e65e736d780\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"
Jul 02 12:40:04.667778 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.667752    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/25db85a8-4af1-4c43-b29d-4e65e736d780-signing-key\") pod \"service-ca-7bb4f4bcc7-twwbr\" (UID: \"25db85a8-4af1-4c43-b29d-4e65e736d780\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"
Jul 02 12:40:04.667889 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.667761    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/25db85a8-4af1-4c43-b29d-4e65e736d780-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-twwbr\" (UID: \"25db85a8-4af1-4c43-b29d-4e65e736d780\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"
Jul 02 12:40:04.673470 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.673441    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76pp2\" (UniqueName: \"kubernetes.io/projected/25db85a8-4af1-4c43-b29d-4e65e736d780-kube-api-access-76pp2\") pod \"service-ca-7bb4f4bcc7-twwbr\" (UID: \"25db85a8-4af1-4c43-b29d-4e65e736d780\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"
Jul 02 12:40:04.767269 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.767240    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"
Jul 02 12:40:04.910750 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:04.910715    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-twwbr"]
Jul 02 12:40:04.915137 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:40:04.915106    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25db85a8_4af1_4c43_b29d_4e65e736d780.slice/crio-a94867ed5d7ed176ad698f07e835f34b789d243310f83bb5c31a53508b028a8d WatchSource:0}: Error finding container a94867ed5d7ed176ad698f07e835f34b789d243310f83bb5c31a53508b028a8d: Status 404 returned error can't find the container with id a94867ed5d7ed176ad698f07e835f34b789d243310f83bb5c31a53508b028a8d
Jul 02 12:40:05.233515 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:05.233489    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pgvs9_f7c2b6aa-6cb9-4b5b-885c-4be0975036ff/node-ca/0.log"
Jul 02 12:40:05.263970 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:05.263940    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:40:05.264108 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:05.264089    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 12:40:05.264159 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:05.264149    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls podName:33aa4b59-fdea-4a0e-8b70-76b57da84a95 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:13.264132259 +0000 UTC m=+61.247605928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-m76wm" (UID: "33aa4b59-fdea-4a0e-8b70-76b57da84a95") : secret "cluster-monitoring-operator-tls" not found
Jul 02 12:40:05.731455 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:05.731418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr" event={"ID":"25db85a8-4af1-4c43-b29d-4e65e736d780","Type":"ContainerStarted","Data":"bd20e38b65f72e0cb02ef4195d1eab2afde752fa22383c172471db9d3f7a016e"}
Jul 02 12:40:05.731836 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:05.731474    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr" event={"ID":"25db85a8-4af1-4c43-b29d-4e65e736d780","Type":"ContainerStarted","Data":"a94867ed5d7ed176ad698f07e835f34b789d243310f83bb5c31a53508b028a8d"}
Jul 02 12:40:05.765407 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:05.765361    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7bb4f4bcc7-twwbr" podStartSLOduration=1.765345043 podStartE2EDuration="1.765345043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:05.765128282 +0000 UTC m=+53.748601967" watchObservedRunningTime="2026-07-02 12:40:05.765345043 +0000 UTC m=+53.748818726"
Jul 02 12:40:05.767447 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:05.767405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:05.767602 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:05.767576    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:40:05.767669 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:05.767645    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls podName:bf13802d-96c8-49d1-b303-ee39802c83f3 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:09.767627697 +0000 UTC m=+57.751101360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ksn6z" (UID: "bf13802d-96c8-49d1-b303-ee39802c83f3") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:40:06.030132 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:06.030032    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-drvvs_0318470e-cccb-4d26-96b4-3b464de83f56/serve-healthcheck-canary/0.log"
Jul 02 12:40:09.800226 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:09.800190    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:09.800806 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:09.800381    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:40:09.800806 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:09.800460    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls podName:bf13802d-96c8-49d1-b303-ee39802c83f3 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:17.800441622 +0000 UTC m=+65.783915288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ksn6z" (UID: "bf13802d-96c8-49d1-b303-ee39802c83f3") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:40:13.332269 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:13.332229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:40:13.332760 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:13.332416    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 12:40:13.332760 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:40:13.332502    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls podName:33aa4b59-fdea-4a0e-8b70-76b57da84a95 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:29.332479908 +0000 UTC m=+77.315953572 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-m76wm" (UID: "33aa4b59-fdea-4a0e-8b70-76b57da84a95") : secret "cluster-monitoring-operator-tls" not found
Jul 02 12:40:13.671245 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:13.671179    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qmx4p"
Jul 02 12:40:17.868239 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:17.868205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:17.870710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:17.870686    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf13802d-96c8-49d1-b303-ee39802c83f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ksn6z\" (UID: \"bf13802d-96c8-49d1-b303-ee39802c83f3\") " pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:17.973207 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:17.973169    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-njqpt\""
Jul 02 12:40:17.980648 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:17.980627    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ksn6z"
Jul 02 12:40:18.112055 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.112014    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ksn6z"]
Jul 02 12:40:18.117617 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:40:18.117590    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf13802d_96c8_49d1_b303_ee39802c83f3.slice/crio-4ac06cd4e8b36bac2cfee915a0b574e368e41e9130ba7307893cfc5cdd2e1435 WatchSource:0}: Error finding container 4ac06cd4e8b36bac2cfee915a0b574e368e41e9130ba7307893cfc5cdd2e1435: Status 404 returned error can't find the container with id 4ac06cd4e8b36bac2cfee915a0b574e368e41e9130ba7307893cfc5cdd2e1435
Jul 02 12:40:18.171529 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.171501    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:40:18.174614 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.174598    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 12:40:18.184202 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.184179    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc387565-3fa1-4271-a12e-dda0c699e3e4-metrics-certs\") pod \"network-metrics-daemon-v7gwd\" (UID: \"dc387565-3fa1-4271-a12e-dda0c699e3e4\") " pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:40:18.271887 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.271845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:40:18.272043 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.271916    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zck\" (UniqueName: \"kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck\") pod \"network-check-target-r975p\" (UID: \"2697b9b4-7477-4e35-a9f3-298ed7a423ed\") " pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:40:18.275852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.275835    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 12:40:18.275917 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.275835    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 12:40:18.285026 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.285000    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1da27b61-6566-4583-a851-61ea1f1e8d11-original-pull-secret\") pod \"global-pull-secret-syncer-bx9gl\" (UID: \"1da27b61-6566-4583-a851-61ea1f1e8d11\") " pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:40:18.286572 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.286559    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 12:40:18.295423 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.295404    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2zck\" (UniqueName: \"kubernetes.io/projected/2697b9b4-7477-4e35-a9f3-298ed7a423ed-kube-api-access-k2zck\") pod \"network-check-target-r975p\" (UID: \"2697b9b4-7477-4e35-a9f3-298ed7a423ed\") " pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:40:18.331223 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.331196    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fnc5f\""
Jul 02 12:40:18.337920 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.337900    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nktrd\""
Jul 02 12:40:18.338924 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.338909    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v7gwd"
Jul 02 12:40:18.342154 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.342139    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx9gl"
Jul 02 12:40:18.346417 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.346401    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:40:18.486262 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.486233    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bx9gl"]
Jul 02 12:40:18.489599 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:40:18.489573    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da27b61_6566_4583_a851_61ea1f1e8d11.slice/crio-7e4aa2362ab612117afd04b9999b0d85dda1063cfc3dc2529074f3dccbedb78d WatchSource:0}: Error finding container 7e4aa2362ab612117afd04b9999b0d85dda1063cfc3dc2529074f3dccbedb78d: Status 404 returned error can't find the container with id 7e4aa2362ab612117afd04b9999b0d85dda1063cfc3dc2529074f3dccbedb78d
Jul 02 12:40:18.724401 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.724380    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r975p"]
Jul 02 12:40:18.726072 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:40:18.726049    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2697b9b4_7477_4e35_a9f3_298ed7a423ed.slice/crio-65726d2c27abeae6ac02112e027b1db8d8249db55fce8ba17a1903a6a11f16c0 WatchSource:0}: Error finding container 65726d2c27abeae6ac02112e027b1db8d8249db55fce8ba17a1903a6a11f16c0: Status 404 returned error can't find the container with id 65726d2c27abeae6ac02112e027b1db8d8249db55fce8ba17a1903a6a11f16c0
Jul 02 12:40:18.728677 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.728598    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v7gwd"]
Jul 02 12:40:18.731889 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:40:18.731868    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc387565_3fa1_4271_a12e_dda0c699e3e4.slice/crio-de8ed760fc58ecf8478d0100c12c4e27679fa34799c6f2794ef52406c6463dfa WatchSource:0}: Error finding container de8ed760fc58ecf8478d0100c12c4e27679fa34799c6f2794ef52406c6463dfa: Status 404 returned error can't find the container with id de8ed760fc58ecf8478d0100c12c4e27679fa34799c6f2794ef52406c6463dfa
Jul 02 12:40:18.766148 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.766113    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r975p" event={"ID":"2697b9b4-7477-4e35-a9f3-298ed7a423ed","Type":"ContainerStarted","Data":"65726d2c27abeae6ac02112e027b1db8d8249db55fce8ba17a1903a6a11f16c0"}
Jul 02 12:40:18.767166 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.767142    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bx9gl" event={"ID":"1da27b61-6566-4583-a851-61ea1f1e8d11","Type":"ContainerStarted","Data":"7e4aa2362ab612117afd04b9999b0d85dda1063cfc3dc2529074f3dccbedb78d"}
Jul 02 12:40:18.768222 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.768201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ksn6z" event={"ID":"bf13802d-96c8-49d1-b303-ee39802c83f3","Type":"ContainerStarted","Data":"4ac06cd4e8b36bac2cfee915a0b574e368e41e9130ba7307893cfc5cdd2e1435"}
Jul 02 12:40:18.769164 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:18.769146    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v7gwd" event={"ID":"dc387565-3fa1-4271-a12e-dda0c699e3e4","Type":"ContainerStarted","Data":"de8ed760fc58ecf8478d0100c12c4e27679fa34799c6f2794ef52406c6463dfa"}
Jul 02 12:40:19.281723 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.281135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:40:19.281723 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.281216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:40:19.285009 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.284980    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d173e052-416f-46c8-a7ed-b3b779c66c66-metrics-tls\") pod \"dns-default-5bz8v\" (UID: \"d173e052-416f-46c8-a7ed-b3b779c66c66\") " pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:40:19.286800 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.286777    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls\") pod \"image-registry-df765d5c7-sj6rs\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") " pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:40:19.465106 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.465076    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s268d\""
Jul 02 12:40:19.472536 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.472513    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:40:19.551511 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.551227    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5bsnw\""
Jul 02 12:40:19.559022 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.558787    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:40:19.657714 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.657575    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-df765d5c7-sj6rs"]
Jul 02 12:40:19.768716 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.768677    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5bz8v"]
Jul 02 12:40:19.773346 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:40:19.773282    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd173e052_416f_46c8_a7ed_b3b779c66c66.slice/crio-6b7aaa3bd9b1ed80ff411437229de498ae3b414412e7ba00f88ccc26cd5e2c2d WatchSource:0}: Error finding container 6b7aaa3bd9b1ed80ff411437229de498ae3b414412e7ba00f88ccc26cd5e2c2d: Status 404 returned error can't find the container with id 6b7aaa3bd9b1ed80ff411437229de498ae3b414412e7ba00f88ccc26cd5e2c2d
Jul 02 12:40:19.784133 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.783164    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-df765d5c7-sj6rs" event={"ID":"3bc1bee4-4948-4f8b-905a-875187f12659","Type":"ContainerStarted","Data":"4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f"}
Jul 02 12:40:19.784133 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.783217    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-df765d5c7-sj6rs" event={"ID":"3bc1bee4-4948-4f8b-905a-875187f12659","Type":"ContainerStarted","Data":"36520e3035ff8785f385aba6e90042a4cfda128de4979fd260412ff5ce6f6884"}
Jul 02 12:40:19.784133 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.784019    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:40:19.812856 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:19.812138    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-df765d5c7-sj6rs" podStartSLOduration=67.812117631 podStartE2EDuration="1m7.812117631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:19.810885009 +0000 UTC m=+67.794358694" watchObservedRunningTime="2026-07-02 12:40:19.812117631 +0000 UTC m=+67.795591319"
Jul 02 12:40:20.789659 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:20.789121    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5bz8v" event={"ID":"d173e052-416f-46c8-a7ed-b3b779c66c66","Type":"ContainerStarted","Data":"6b7aaa3bd9b1ed80ff411437229de498ae3b414412e7ba00f88ccc26cd5e2c2d"}
Jul 02 12:40:23.858582 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.858551    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n"]
Jul 02 12:40:23.860728 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.860702    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n"
Jul 02 12:40:23.865522 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.865498    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-gtt58\""
Jul 02 12:40:23.866577 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.866211    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 02 12:40:23.866577 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.866346    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 02 12:40:23.866577 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.866441    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 02 12:40:23.866917 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.866900    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 02 12:40:23.910926 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.910895    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n"]
Jul 02 12:40:23.932217 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.932180    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tns2l\" (UniqueName: \"kubernetes.io/projected/4935ebdb-4122-46d7-85ae-6d8db67e25c2-kube-api-access-tns2l\") pod \"managed-serviceaccount-addon-agent-755df96546-jk45n\" (UID: \"4935ebdb-4122-46d7-85ae-6d8db67e25c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n"
Jul 02 12:40:23.932399 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.932241    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4935ebdb-4122-46d7-85ae-6d8db67e25c2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-755df96546-jk45n\" (UID: \"4935ebdb-4122-46d7-85ae-6d8db67e25c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n"
Jul 02 12:40:23.982546 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.982512    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49"]
Jul 02 12:40:23.984574 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.984546    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49"
Jul 02 12:40:23.989410 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.989358    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7z6xf\""
Jul 02 12:40:23.989681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.989653    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 02 12:40:23.990275 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:23.990248    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 02 12:40:24.033621 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.033582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns2l\" (UniqueName: \"kubernetes.io/projected/4935ebdb-4122-46d7-85ae-6d8db67e25c2-kube-api-access-tns2l\") pod \"managed-serviceaccount-addon-agent-755df96546-jk45n\" (UID: \"4935ebdb-4122-46d7-85ae-6d8db67e25c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n"
Jul 02 12:40:24.033621 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.033624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4935ebdb-4122-46d7-85ae-6d8db67e25c2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-755df96546-jk45n\" (UID: \"4935ebdb-4122-46d7-85ae-6d8db67e25c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n"
Jul 02 12:40:24.036390 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.036361    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4935ebdb-4122-46d7-85ae-6d8db67e25c2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-755df96546-jk45n\" (UID: \"4935ebdb-4122-46d7-85ae-6d8db67e25c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n"
Jul 02 12:40:24.066075 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.066035    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tns2l\" (UniqueName: \"kubernetes.io/projected/4935ebdb-4122-46d7-85ae-6d8db67e25c2-kube-api-access-tns2l\") pod \"managed-serviceaccount-addon-agent-755df96546-jk45n\" (UID: \"4935ebdb-4122-46d7-85ae-6d8db67e25c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n"
Jul 02 12:40:24.079723 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.079689    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49"]
Jul 02 12:40:24.134238 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.134150    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34416683-a3ce-4dbd-9b1b-b6f1d781b90a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-8rg49\" (UID: \"34416683-a3ce-4dbd-9b1b-b6f1d781b90a\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49"
Jul 02 12:40:24.134415 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.134256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34416683-a3ce-4dbd-9b1b-b6f1d781b90a-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-8rg49\" (UID: \"34416683-a3ce-4dbd-9b1b-b6f1d781b90a\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49"
Jul 02 12:40:24.194264 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.194231    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n"
Jul 02 12:40:24.235579 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.235549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34416683-a3ce-4dbd-9b1b-b6f1d781b90a-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-8rg49\" (UID: \"34416683-a3ce-4dbd-9b1b-b6f1d781b90a\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49"
Jul 02 12:40:24.235750 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.235605    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34416683-a3ce-4dbd-9b1b-b6f1d781b90a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-8rg49\" (UID: \"34416683-a3ce-4dbd-9b1b-b6f1d781b90a\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49"
Jul 02 12:40:24.236288 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.236266    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34416683-a3ce-4dbd-9b1b-b6f1d781b90a-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-8rg49\" (UID: \"34416683-a3ce-4dbd-9b1b-b6f1d781b90a\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49"
Jul 02 12:40:24.238159 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.238139    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34416683-a3ce-4dbd-9b1b-b6f1d781b90a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-8rg49\" (UID: \"34416683-a3ce-4dbd-9b1b-b6f1d781b90a\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49"
Jul 02 12:40:24.295340 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:24.295291    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49"
Jul 02 12:40:26.771534 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:26.771483    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n"]
Jul 02 12:40:26.813408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:26.813334    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49"]
Jul 02 12:40:26.818835 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:40:26.818807    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34416683_a3ce_4dbd_9b1b_b6f1d781b90a.slice/crio-869e9f4318a1cfec6469395c0cbbcfb2db27318cd4d30132595340af1a619ae2 WatchSource:0}: Error finding container 869e9f4318a1cfec6469395c0cbbcfb2db27318cd4d30132595340af1a619ae2: Status 404 returned error can't find the container with id 869e9f4318a1cfec6469395c0cbbcfb2db27318cd4d30132595340af1a619ae2
Jul 02 12:40:26.819724 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:26.819652    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n" event={"ID":"4935ebdb-4122-46d7-85ae-6d8db67e25c2","Type":"ContainerStarted","Data":"fd1da2321381052d2b17c3b84769bda82cef6834ae8e9d7d7e2f2f62e2a89a49"}
Jul 02 12:40:26.831460 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:26.826238    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v7gwd" event={"ID":"dc387565-3fa1-4271-a12e-dda0c699e3e4","Type":"ContainerStarted","Data":"ce5a650848b6d056772f041a76bde06a5bbfd682b7994441dd14a98dea01fbc1"}
Jul 02 12:40:26.832857 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:26.832464    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r975p" event={"ID":"2697b9b4-7477-4e35-a9f3-298ed7a423ed","Type":"ContainerStarted","Data":"d22cdfa0b655d303f4f709892611eed65050fbec5358889cd88e993d8f5b2b2e"}
Jul 02 12:40:26.832857 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:26.832803    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:40:26.835798 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:26.835684    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bx9gl" event={"ID":"1da27b61-6566-4583-a851-61ea1f1e8d11","Type":"ContainerStarted","Data":"60c4b979cca8dd81b48dce6ad2cdfe2ab104e6ddc9607acbbc9ef161e90a2a6c"}
Jul 02 12:40:26.840231 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:26.840203    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ksn6z" event={"ID":"bf13802d-96c8-49d1-b303-ee39802c83f3","Type":"ContainerStarted","Data":"3d52ec48837dec1cd7091c96620b8524d272a9fc73c45b6bf1b25bfa6e187cd8"}
Jul 02 12:40:26.852711 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:26.851988    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r975p" podStartSLOduration=67.447061157 podStartE2EDuration="1m14.851973006s" totalImagesPullingTime="7.404911849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:26.851498444 +0000 UTC m=+74.834972128" watchObservedRunningTime="2026-07-02 12:40:26.851973006 +0000 UTC m=+74.835446691"
Jul 02 12:40:26.872713 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:26.872337    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bx9gl" podStartSLOduration=66.727488727 podStartE2EDuration="1m14.872315516s" totalImagesPullingTime="8.144826789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:26.871613879 +0000 UTC m=+74.855087567" watchObservedRunningTime="2026-07-02 12:40:26.872315516 +0000 UTC m=+74.855789200"
Jul 02 12:40:27.849732 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:27.849697    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ksn6z" event={"ID":"bf13802d-96c8-49d1-b303-ee39802c83f3","Type":"ContainerStarted","Data":"5d132eae185e53062df0c768bf4895005fe56ff41239441ff74b5b67892e34d6"}
Jul 02 12:40:27.852075 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:27.852047    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5bz8v" event={"ID":"d173e052-416f-46c8-a7ed-b3b779c66c66","Type":"ContainerStarted","Data":"2bfc1ab785513aecee7c3df1e51107b4378a5d52b46016fef904f2290d78c54e"}
Jul 02 12:40:27.852200 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:27.852102    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5bz8v" event={"ID":"d173e052-416f-46c8-a7ed-b3b779c66c66","Type":"ContainerStarted","Data":"f0a9b53e59d821d356564658e564ba81cb588ee3af4e3537c8e546afccc24ac7"}
Jul 02 12:40:27.852581 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:27.852366    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:40:27.854078 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:27.854016    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49" event={"ID":"34416683-a3ce-4dbd-9b1b-b6f1d781b90a","Type":"ContainerStarted","Data":"869e9f4318a1cfec6469395c0cbbcfb2db27318cd4d30132595340af1a619ae2"}
Jul 02 12:40:27.857396 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:27.857373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v7gwd" event={"ID":"dc387565-3fa1-4271-a12e-dda0c699e3e4","Type":"ContainerStarted","Data":"70c7e6b6ccafa5df02a346763264f2cf92aa5af2e0625dda70186c92a4ea8b16"}
Jul 02 12:40:27.874351 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:27.874287    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5bz8v" podStartSLOduration=34.008202572 podStartE2EDuration="40.874268105s" totalImagesPullingTime="6.866065533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:27.872344139 +0000 UTC m=+75.855817824" watchObservedRunningTime="2026-07-02 12:40:27.874268105 +0000 UTC m=+75.857741791"
Jul 02 12:40:27.893943 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:27.893714    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v7gwd" podStartSLOduration=68.828942844 podStartE2EDuration="1m15.893698974s" totalImagesPullingTime="7.06475613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:27.893390492 +0000 UTC m=+75.876864177" watchObservedRunningTime="2026-07-02 12:40:27.893698974 +0000 UTC m=+75.877172659"
Jul 02 12:40:29.382877 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:29.382834    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:40:29.386026 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:29.385992    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33aa4b59-fdea-4a0e-8b70-76b57da84a95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-m76wm\" (UID: \"33aa4b59-fdea-4a0e-8b70-76b57da84a95\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:40:29.638787 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:29.638707    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8xhgv\""
Jul 02 12:40:29.646172 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:29.646147    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"
Jul 02 12:40:30.710498 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:30.710466    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm"]
Jul 02 12:40:30.713152 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:40:30.713126    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33aa4b59_fdea_4a0e_8b70_76b57da84a95.slice/crio-9c3cfb9ba8309c7cd17a016d29d67f8e7eb10e29bd3415e9c622bfceb5f418ce WatchSource:0}: Error finding container 9c3cfb9ba8309c7cd17a016d29d67f8e7eb10e29bd3415e9c622bfceb5f418ce: Status 404 returned error can't find the container with id 9c3cfb9ba8309c7cd17a016d29d67f8e7eb10e29bd3415e9c622bfceb5f418ce
Jul 02 12:40:30.870378 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:30.870279    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ksn6z" event={"ID":"bf13802d-96c8-49d1-b303-ee39802c83f3","Type":"ContainerStarted","Data":"39779647dc186de5279503cf35672fe2a0f9d030cdbde55388ae2b089b3e7769"}
Jul 02 12:40:30.871583 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:30.871559    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm" event={"ID":"33aa4b59-fdea-4a0e-8b70-76b57da84a95","Type":"ContainerStarted","Data":"9c3cfb9ba8309c7cd17a016d29d67f8e7eb10e29bd3415e9c622bfceb5f418ce"}
Jul 02 12:40:30.872938 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:30.872916    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n" event={"ID":"4935ebdb-4122-46d7-85ae-6d8db67e25c2","Type":"ContainerStarted","Data":"ea21c9c929c945ea7cae2691c4f4846b75f97dcc25194ac547160af86035d004"}
Jul 02 12:40:30.874348 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:30.874307    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49" event={"ID":"34416683-a3ce-4dbd-9b1b-b6f1d781b90a","Type":"ContainerStarted","Data":"ef50f60e0f7d8c784d4e53efc66b41f75e9a7e87064bb331e8984a9e9e507704"}
Jul 02 12:40:30.890182 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:30.890138    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ksn6z" podStartSLOduration=16.642481481 podStartE2EDuration="28.890123836s" totalImagesPullingTime="12.247642355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:40:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:30.889388317 +0000 UTC m=+78.872862003" watchObservedRunningTime="2026-07-02 12:40:30.890123836 +0000 UTC m=+78.873597519"
Jul 02 12:40:30.907062 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:30.907016    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755df96546-jk45n" podStartSLOduration=4.141232486 podStartE2EDuration="7.907002997s" totalImagesPullingTime="3.765770511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:30.905714699 +0000 UTC m=+78.889188381" watchObservedRunningTime="2026-07-02 12:40:30.907002997 +0000 UTC m=+78.890476679"
Jul 02 12:40:30.925802 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:30.925752    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-8rg49" podStartSLOduration=4.215291018 podStartE2EDuration="7.925738633s" totalImagesPullingTime="3.710447615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:30.925342809 +0000 UTC m=+78.908816493" watchObservedRunningTime="2026-07-02 12:40:30.925738633 +0000 UTC m=+78.909212294"
Jul 02 12:40:32.574644 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.574611    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79695b9df5-zhmzj"]
Jul 02 12:40:32.579583 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.579558    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.582480 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.582457    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 12:40:32.582480 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.582457    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 12:40:32.585040 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.584620    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 12:40:32.585040 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.584655    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 12:40:32.585040 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.584890    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 12:40:32.585040 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.584977    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 12:40:32.585346 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.585242    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 12:40:32.585346 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.585339    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2kq2r\""
Jul 02 12:40:32.590068 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.590041    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79695b9df5-zhmzj"]
Jul 02 12:40:32.706120 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.706083    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f392a7a7-9314-471e-8c66-05658289559e-console-oauth-config\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.706120 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.706124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-oauth-serving-cert\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.706342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.706205    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lftfs\" (UniqueName: \"kubernetes.io/projected/f392a7a7-9314-471e-8c66-05658289559e-kube-api-access-lftfs\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.706342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.706255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f392a7a7-9314-471e-8c66-05658289559e-console-serving-cert\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.706342 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.706291    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-console-config\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.706469 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.706354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-service-ca\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.807040 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.807005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lftfs\" (UniqueName: \"kubernetes.io/projected/f392a7a7-9314-471e-8c66-05658289559e-kube-api-access-lftfs\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.807170 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.807051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f392a7a7-9314-471e-8c66-05658289559e-console-serving-cert\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.807170 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.807072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-console-config\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.807170 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.807093    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-service-ca\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.807339 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.807274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f392a7a7-9314-471e-8c66-05658289559e-console-oauth-config\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.807339 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.807328    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-oauth-serving-cert\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.807721 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.807700    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-service-ca\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.807848 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.807832    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-console-config\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.807938 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.807923    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-oauth-serving-cert\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.809317 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.809287    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f392a7a7-9314-471e-8c66-05658289559e-console-serving-cert\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.809453 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.809438    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f392a7a7-9314-471e-8c66-05658289559e-console-oauth-config\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.819407 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.819382    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lftfs\" (UniqueName: \"kubernetes.io/projected/f392a7a7-9314-471e-8c66-05658289559e-kube-api-access-lftfs\") pod \"console-79695b9df5-zhmzj\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") " pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.883888 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.883851    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm" event={"ID":"33aa4b59-fdea-4a0e-8b70-76b57da84a95","Type":"ContainerStarted","Data":"da12d3743e48d252beaeee91330861e4ad79b64bfb785420a58e73a9de10a669"}
Jul 02 12:40:32.891768 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.891740    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:32.903970 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:32.903914    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-m76wm" podStartSLOduration=33.839131772 podStartE2EDuration="35.903894657s" totalImagesPullingTime="2.064762885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:32.903205809 +0000 UTC m=+80.886679494" watchObservedRunningTime="2026-07-02 12:40:32.903894657 +0000 UTC m=+80.887368343"
Jul 02 12:40:33.031454 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:33.031423    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79695b9df5-zhmzj"]
Jul 02 12:40:33.034522 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:40:33.034495    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf392a7a7_9314_471e_8c66_05658289559e.slice/crio-7ac85ea8c5e265c1fe351b34069dcd828f99cd97c7cba92937c05136bad6dd4d WatchSource:0}: Error finding container 7ac85ea8c5e265c1fe351b34069dcd828f99cd97c7cba92937c05136bad6dd4d: Status 404 returned error can't find the container with id 7ac85ea8c5e265c1fe351b34069dcd828f99cd97c7cba92937c05136bad6dd4d
Jul 02 12:40:33.889048 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:33.889005    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79695b9df5-zhmzj" event={"ID":"f392a7a7-9314-471e-8c66-05658289559e","Type":"ContainerStarted","Data":"7ac85ea8c5e265c1fe351b34069dcd828f99cd97c7cba92937c05136bad6dd4d"}
Jul 02 12:40:36.900284 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:36.900250    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79695b9df5-zhmzj" event={"ID":"f392a7a7-9314-471e-8c66-05658289559e","Type":"ContainerStarted","Data":"505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54"}
Jul 02 12:40:36.919407 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:36.919184    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79695b9df5-zhmzj" podStartSLOduration=2.108500566 podStartE2EDuration="4.91916242s" totalImagesPullingTime="2.810661854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:36.918194316 +0000 UTC m=+84.901667996" watchObservedRunningTime="2026-07-02 12:40:36.91916242 +0000 UTC m=+84.902636103"
Jul 02 12:40:37.864218 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:37.864188    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5bz8v"
Jul 02 12:40:41.168434 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.168404    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f698b95d6-j9vvs"]
Jul 02 12:40:41.172698 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.172682    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.181869 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.181852    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 02 12:40:41.188407 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.188384    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f698b95d6-j9vvs"]
Jul 02 12:40:41.277607 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.277571    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-service-ca\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.277607 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.277609    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-serving-cert\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.277844 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.277631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-oauth-serving-cert\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.277844 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.277652    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-config\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.277844 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.277725    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-oauth-config\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.277844 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.277757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-trusted-ca-bundle\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.277844 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.277786    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgxtd\" (UniqueName: \"kubernetes.io/projected/9b5fec4e-e798-4944-9cbe-0197acb12b32-kube-api-access-dgxtd\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.378996 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.378954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgxtd\" (UniqueName: \"kubernetes.io/projected/9b5fec4e-e798-4944-9cbe-0197acb12b32-kube-api-access-dgxtd\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.378996 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.379004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-service-ca\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.379235 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.379030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-serving-cert\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.379235 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.379055    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-oauth-serving-cert\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.379235 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.379079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-config\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.379235 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.379112    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-oauth-config\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.379463 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.379235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-trusted-ca-bundle\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.379952 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.379913    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-oauth-serving-cert\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.379952 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.379938    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-config\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.380075 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.379947    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-service-ca\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.380075 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.380051    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-trusted-ca-bundle\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.381598 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.381577    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-oauth-config\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.381680 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.381642    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-serving-cert\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.387422 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.387403    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgxtd\" (UniqueName: \"kubernetes.io/projected/9b5fec4e-e798-4944-9cbe-0197acb12b32-kube-api-access-dgxtd\") pod \"console-7f698b95d6-j9vvs\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") " pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.482144 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.482105    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:41.610937 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.610904    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f698b95d6-j9vvs"]
Jul 02 12:40:41.614206 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:40:41.614181    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5fec4e_e798_4944_9cbe_0197acb12b32.slice/crio-a7f317929a1f936a347dd6b517a5055f68ca20a9f06d8a434c28d698884abcef WatchSource:0}: Error finding container a7f317929a1f936a347dd6b517a5055f68ca20a9f06d8a434c28d698884abcef: Status 404 returned error can't find the container with id a7f317929a1f936a347dd6b517a5055f68ca20a9f06d8a434c28d698884abcef
Jul 02 12:40:41.796838 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.796758    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:40:41.918376 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.918344    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f698b95d6-j9vvs" event={"ID":"9b5fec4e-e798-4944-9cbe-0197acb12b32","Type":"ContainerStarted","Data":"18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7"}
Jul 02 12:40:41.918576 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.918393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f698b95d6-j9vvs" event={"ID":"9b5fec4e-e798-4944-9cbe-0197acb12b32","Type":"ContainerStarted","Data":"a7f317929a1f936a347dd6b517a5055f68ca20a9f06d8a434c28d698884abcef"}
Jul 02 12:40:41.939340 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:41.939274    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f698b95d6-j9vvs" podStartSLOduration=0.939259808 podStartE2EDuration="939.259808ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:41.938061665 +0000 UTC m=+89.921535350" watchObservedRunningTime="2026-07-02 12:40:41.939259808 +0000 UTC m=+89.922733490"
Jul 02 12:40:42.891968 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:42.891940    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:42.892421 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:42.892165    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:42.896636 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:42.896613    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:42.926147 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:42.926121    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:40:47.669943 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:47.669909    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-df765d5c7-sj6rs"]
Jul 02 12:40:51.482813 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.482778    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:51.483231 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.482868    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:51.487984 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.487957    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:51.789666 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.789575    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tqzrs"]
Jul 02 12:40:51.794169 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.794145    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.796824 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.796779    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 12:40:51.798152 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.798129    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 12:40:51.798346 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.798322    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rsb9g\""
Jul 02 12:40:51.798497 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.798480    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 12:40:51.798639 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.798623    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 12:40:51.852695 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.852661    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.852695 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.852693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-wtmp\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.852909 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.852712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-textfile\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.852909 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.852739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-tls\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.852909 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.852789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a539811f-cf06-446c-b295-14b8f63ea5ed-sys\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.852909 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.852823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.852909 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.852850    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a539811f-cf06-446c-b295-14b8f63ea5ed-metrics-client-ca\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.852909 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.852876    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a539811f-cf06-446c-b295-14b8f63ea5ed-root\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.853245 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.852952    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z7m2\" (UniqueName: \"kubernetes.io/projected/a539811f-cf06-446c-b295-14b8f63ea5ed-kube-api-access-6z7m2\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.953858 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.953825    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-tls\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.953858 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.953863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a539811f-cf06-446c-b295-14b8f63ea5ed-sys\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.954153 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.953880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.954153 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.953908    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a539811f-cf06-446c-b295-14b8f63ea5ed-metrics-client-ca\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.954153 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.953937    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a539811f-cf06-446c-b295-14b8f63ea5ed-root\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.954153 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.953945    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a539811f-cf06-446c-b295-14b8f63ea5ed-sys\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.954153 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.953967    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z7m2\" (UniqueName: \"kubernetes.io/projected/a539811f-cf06-446c-b295-14b8f63ea5ed-kube-api-access-6z7m2\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.954153 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.954004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.954153 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.954023    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-wtmp\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.954153 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.954039    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-textfile\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.954579 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.954364    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-textfile\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.954579 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.954506    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-wtmp\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.954754 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.954730    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a539811f-cf06-446c-b295-14b8f63ea5ed-root\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.954924 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.954745    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a539811f-cf06-446c-b295-14b8f63ea5ed-metrics-client-ca\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.955021 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.954951    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.956610 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.956592    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-tls\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.956871 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.956855    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a539811f-cf06-446c-b295-14b8f63ea5ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:51.959838 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.959817    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:40:51.968180 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:51.968159    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z7m2\" (UniqueName: \"kubernetes.io/projected/a539811f-cf06-446c-b295-14b8f63ea5ed-kube-api-access-6z7m2\") pod \"node-exporter-tqzrs\" (UID: \"a539811f-cf06-446c-b295-14b8f63ea5ed\") " pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:52.020609 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.020575    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79695b9df5-zhmzj"]
Jul 02 12:40:52.105171 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.105089    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqzrs"
Jul 02 12:40:52.122873 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:40:52.122842    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda539811f_cf06_446c_b295_14b8f63ea5ed.slice/crio-339852c6c87d69b3224ee0beb5ed46ca43e40f29bcd869c020deccc233977201 WatchSource:0}: Error finding container 339852c6c87d69b3224ee0beb5ed46ca43e40f29bcd869c020deccc233977201: Status 404 returned error can't find the container with id 339852c6c87d69b3224ee0beb5ed46ca43e40f29bcd869c020deccc233977201
Jul 02 12:40:52.930004 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.929965    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 12:40:52.935359 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.935330    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.942831 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.942802    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 02 12:40:52.942952 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.942891    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 02 12:40:52.952230 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.952208    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 02 12:40:52.952475 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.952456    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 02 12:40:52.952590 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.952551    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 02 12:40:52.952677 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.952662    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 02 12:40:52.952741 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.952669    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 02 12:40:52.952800 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.952787    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 02 12:40:52.952857 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.952792    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jlh7n\""
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961111    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961317    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6f21957-0de2-44f3-9625-5e208e2bb4be-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961349    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6f21957-0de2-44f3-9625-5e208e2bb4be-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961525    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-config-volume\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961548    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-web-config\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961610    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f21957-0de2-44f3-9625-5e208e2bb4be-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961637    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqsvf\" (UniqueName: \"kubernetes.io/projected/f6f21957-0de2-44f3-9625-5e208e2bb4be-kube-api-access-zqsvf\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961768    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961794    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6f21957-0de2-44f3-9625-5e208e2bb4be-config-out\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.963124 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.961837    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6f21957-0de2-44f3-9625-5e208e2bb4be-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:52.966219 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.966180    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 12:40:52.966351 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.966234    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 02 12:40:52.967288 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:52.966928    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqzrs" event={"ID":"a539811f-cf06-446c-b295-14b8f63ea5ed","Type":"ContainerStarted","Data":"339852c6c87d69b3224ee0beb5ed46ca43e40f29bcd869c020deccc233977201"}
Jul 02 12:40:53.063156 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-config-volume\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.063236 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063172    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-web-config\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.063236 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063202    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.063358 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f21957-0de2-44f3-9625-5e208e2bb4be-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.063358 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqsvf\" (UniqueName: \"kubernetes.io/projected/f6f21957-0de2-44f3-9625-5e208e2bb4be-kube-api-access-zqsvf\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.063461 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.063461 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6f21957-0de2-44f3-9625-5e208e2bb4be-config-out\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.063559 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6f21957-0de2-44f3-9625-5e208e2bb4be-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.063609 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.063660 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.063715 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6f21957-0de2-44f3-9625-5e208e2bb4be-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.063715 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063708    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6f21957-0de2-44f3-9625-5e208e2bb4be-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.063802 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.063737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.065700 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.064656    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6f21957-0de2-44f3-9625-5e208e2bb4be-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.065700 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.065319    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6f21957-0de2-44f3-9625-5e208e2bb4be-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.065861 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.065733    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f21957-0de2-44f3-9625-5e208e2bb4be-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.066694 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.066668    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.069118 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.069025    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6f21957-0de2-44f3-9625-5e208e2bb4be-config-out\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.070193 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.070151    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.070291 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.070203    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.070291 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.070252    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6f21957-0de2-44f3-9625-5e208e2bb4be-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.070423 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.070391    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-config-volume\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.070593 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.070572    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.071573 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.071553    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.071843 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.071825    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6f21957-0de2-44f3-9625-5e208e2bb4be-web-config\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.082613 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.082589    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqsvf\" (UniqueName: \"kubernetes.io/projected/f6f21957-0de2-44f3-9625-5e208e2bb4be-kube-api-access-zqsvf\") pod \"alertmanager-main-0\" (UID: \"f6f21957-0de2-44f3-9625-5e208e2bb4be\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.248311 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.248277    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:40:53.419607 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.419237    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 12:40:53.424063 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:40:53.424040    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f21957_0de2_44f3_9625_5e208e2bb4be.slice/crio-5214a2695b082f8afb8ab6c93dc617c53255ca593708aad8810926e35fe98b27 WatchSource:0}: Error finding container 5214a2695b082f8afb8ab6c93dc617c53255ca593708aad8810926e35fe98b27: Status 404 returned error can't find the container with id 5214a2695b082f8afb8ab6c93dc617c53255ca593708aad8810926e35fe98b27
Jul 02 12:40:53.971260 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.971219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6f21957-0de2-44f3-9625-5e208e2bb4be","Type":"ContainerStarted","Data":"5214a2695b082f8afb8ab6c93dc617c53255ca593708aad8810926e35fe98b27"}
Jul 02 12:40:53.972840 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.972814    2564 generic.go:356] "Generic (PLEG): container finished" podID="a539811f-cf06-446c-b295-14b8f63ea5ed" containerID="5650470d04c56e4c2cedbd54552b52951f76d8de303d1e8eb0a6b9051bdb6000" exitCode=0
Jul 02 12:40:53.972955 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:53.972898    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqzrs" event={"ID":"a539811f-cf06-446c-b295-14b8f63ea5ed","Type":"ContainerDied","Data":"5650470d04c56e4c2cedbd54552b52951f76d8de303d1e8eb0a6b9051bdb6000"}
Jul 02 12:40:54.978483 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:54.978400    2564 generic.go:356] "Generic (PLEG): container finished" podID="f6f21957-0de2-44f3-9625-5e208e2bb4be" containerID="e8d95b508a25c6853c976d161fff420e2b3ac790e01808e79d57f51b775a8720" exitCode=0
Jul 02 12:40:54.978874 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:54.978487    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6f21957-0de2-44f3-9625-5e208e2bb4be","Type":"ContainerDied","Data":"e8d95b508a25c6853c976d161fff420e2b3ac790e01808e79d57f51b775a8720"}
Jul 02 12:40:54.980993 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:54.980968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqzrs" event={"ID":"a539811f-cf06-446c-b295-14b8f63ea5ed","Type":"ContainerStarted","Data":"0f6a1363ba6b87fde9d569407aed102b8970a811f31b02eacf067e4dfa553f8a"}
Jul 02 12:40:54.981098 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:54.981011    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqzrs" event={"ID":"a539811f-cf06-446c-b295-14b8f63ea5ed","Type":"ContainerStarted","Data":"8070321b1f3826708e37df3a57f297d520b4a3bba4d31668afff11c7fa24e032"}
Jul 02 12:40:55.117924 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:55.114347    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tqzrs" podStartSLOduration=3.129709178 podStartE2EDuration="4.11432769s" totalImagesPullingTime="920.198089ms" totalInitContainerRuntime="64.420423ms" isStatefulPod=true podCreationTimestamp="2026-07-02 12:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:55.112459743 +0000 UTC m=+103.095933426" watchObservedRunningTime="2026-07-02 12:40:55.11432769 +0000 UTC m=+103.097801375"
Jul 02 12:40:57.861120 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:57.861099    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r975p"
Jul 02 12:40:57.996089 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:57.996057    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6f21957-0de2-44f3-9625-5e208e2bb4be","Type":"ContainerStarted","Data":"e8b5dcc9e99caa1fdfb0db38b4302d2a4d744809b697cc4d474496e54e60324b"}
Jul 02 12:40:57.996436 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:57.996128    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6f21957-0de2-44f3-9625-5e208e2bb4be","Type":"ContainerStarted","Data":"675af1803bab42680b1cb79b13b3f7ff64e07ded489ab8d0f719cf788c8c4b7e"}
Jul 02 12:40:57.996436 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:57.996155    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6f21957-0de2-44f3-9625-5e208e2bb4be","Type":"ContainerStarted","Data":"f6ad0bdce8f4e31a1c6dc764a3b630f3ddd5c41a3ddb4898feaa7787940ec1b5"}
Jul 02 12:40:57.996436 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:57.996185    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6f21957-0de2-44f3-9625-5e208e2bb4be","Type":"ContainerStarted","Data":"d0b7f1de53159ddecf0b9e525a1c89ae0e0ac7eaec2368c8f9b5cd25ed79fe78"}
Jul 02 12:40:57.996436 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:57.996209    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6f21957-0de2-44f3-9625-5e208e2bb4be","Type":"ContainerStarted","Data":"e2368f41b0d5ba60acb275a55ae63d5b58d1bcf88b0607b2f8462f9da11e2b14"}
Jul 02 12:40:58.225910 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.225878    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77fc79b6fb-wk7w2"]
Jul 02 12:40:58.253013 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.252886    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77fc79b6fb-wk7w2"]
Jul 02 12:40:58.253013 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.252999    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.305654 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.305619    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-oauth-serving-cert\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.305835 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.305661    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-trusted-ca-bundle\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.305835 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.305699    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-console-config\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.305835 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.305724    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92fd9b58-5e07-4999-9762-7c329becd682-console-oauth-config\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.305835 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.305806    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5th6r\" (UniqueName: \"kubernetes.io/projected/92fd9b58-5e07-4999-9762-7c329becd682-kube-api-access-5th6r\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.305982 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.305853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92fd9b58-5e07-4999-9762-7c329becd682-console-serving-cert\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.305982 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.305869    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-service-ca\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.407260 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.407224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-trusted-ca-bundle\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.407485 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.407272    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-console-config\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.407485 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.407291    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92fd9b58-5e07-4999-9762-7c329becd682-console-oauth-config\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.407485 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.407348    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5th6r\" (UniqueName: \"kubernetes.io/projected/92fd9b58-5e07-4999-9762-7c329becd682-kube-api-access-5th6r\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.407485 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.407385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92fd9b58-5e07-4999-9762-7c329becd682-console-serving-cert\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.407485 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.407402    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-service-ca\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.407485 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.407443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-oauth-serving-cert\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.408093 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.408064    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-service-ca\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.408093 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.408088    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-oauth-serving-cert\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.408241 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.408096    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-console-config\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.408579 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.408550    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-trusted-ca-bundle\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.409811 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.409787    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92fd9b58-5e07-4999-9762-7c329becd682-console-oauth-config\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.410337 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.410318    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92fd9b58-5e07-4999-9762-7c329becd682-console-serving-cert\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.417142 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.417121    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5th6r\" (UniqueName: \"kubernetes.io/projected/92fd9b58-5e07-4999-9762-7c329becd682-kube-api-access-5th6r\") pod \"console-77fc79b6fb-wk7w2\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") " pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.561992 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.561912    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:40:58.724997 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:58.722276    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77fc79b6fb-wk7w2"]
Jul 02 12:40:59.002007 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:59.001968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77fc79b6fb-wk7w2" event={"ID":"92fd9b58-5e07-4999-9762-7c329becd682","Type":"ContainerStarted","Data":"8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136"}
Jul 02 12:40:59.002389 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:59.002020    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77fc79b6fb-wk7w2" event={"ID":"92fd9b58-5e07-4999-9762-7c329becd682","Type":"ContainerStarted","Data":"d0bd6be30b70413952d3e0c2c5281708ed229b0a4666f6250fa74aa0b3d8b0df"}
Jul 02 12:40:59.026388 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:40:59.026337    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77fc79b6fb-wk7w2" podStartSLOduration=1.026320697 podStartE2EDuration="1.026320697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:59.024923294 +0000 UTC m=+107.008396979" watchObservedRunningTime="2026-07-02 12:40:59.026320697 +0000 UTC m=+107.009794431"
Jul 02 12:41:00.011232 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:00.011186    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6f21957-0de2-44f3-9625-5e208e2bb4be","Type":"ContainerStarted","Data":"51b1489d0d3a46a69506be1e89f0101c331bfbe99af0dea103ee7f21ce0f63a7"}
Jul 02 12:41:00.063033 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:00.062980    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.69159377 podStartE2EDuration="8.062963886s" totalImagesPullingTime="5.3505935s" totalInitContainerRuntime="20.776616ms" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:41:00.061273968 +0000 UTC m=+108.044747652" watchObservedRunningTime="2026-07-02 12:41:00.062963886 +0000 UTC m=+108.046437568"
Jul 02 12:41:08.562790 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:08.562755    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:41:08.562790 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:08.562802    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:41:08.567803 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:08.567775    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:41:09.050457 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:09.050426    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:41:09.102094 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:09.102067    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f698b95d6-j9vvs"]
Jul 02 12:41:12.696043 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:12.695990    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-df765d5c7-sj6rs" podUID="3bc1bee4-4948-4f8b-905a-875187f12659" containerName="registry" containerID="cri-o://4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f" gracePeriod=30
Jul 02 12:41:12.935737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:12.935708    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:41:13.033742 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.033664    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls\") pod \"3bc1bee4-4948-4f8b-905a-875187f12659\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") "
Jul 02 12:41:13.033742 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.033701    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bc1bee4-4948-4f8b-905a-875187f12659-trusted-ca\") pod \"3bc1bee4-4948-4f8b-905a-875187f12659\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") "
Jul 02 12:41:13.033742 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.033725    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bc1bee4-4948-4f8b-905a-875187f12659-registry-certificates\") pod \"3bc1bee4-4948-4f8b-905a-875187f12659\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") "
Jul 02 12:41:13.033950 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.033866    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bc1bee4-4948-4f8b-905a-875187f12659-image-registry-private-configuration\") pod \"3bc1bee4-4948-4f8b-905a-875187f12659\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") "
Jul 02 12:41:13.033950 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.033926    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-bound-sa-token\") pod \"3bc1bee4-4948-4f8b-905a-875187f12659\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") "
Jul 02 12:41:13.034038 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.033949    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bc1bee4-4948-4f8b-905a-875187f12659-installation-pull-secrets\") pod \"3bc1bee4-4948-4f8b-905a-875187f12659\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") "
Jul 02 12:41:13.034038 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.034013    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5djp5\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-kube-api-access-5djp5\") pod \"3bc1bee4-4948-4f8b-905a-875187f12659\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") "
Jul 02 12:41:13.034136 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.034043    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bc1bee4-4948-4f8b-905a-875187f12659-ca-trust-extracted\") pod \"3bc1bee4-4948-4f8b-905a-875187f12659\" (UID: \"3bc1bee4-4948-4f8b-905a-875187f12659\") "
Jul 02 12:41:13.034189 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.034132    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc1bee4-4948-4f8b-905a-875187f12659-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3bc1bee4-4948-4f8b-905a-875187f12659" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:41:13.034404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.034358    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bc1bee4-4948-4f8b-905a-875187f12659-registry-certificates\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:13.034404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.034246    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc1bee4-4948-4f8b-905a-875187f12659-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3bc1bee4-4948-4f8b-905a-875187f12659" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:41:13.036432 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.036402    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3bc1bee4-4948-4f8b-905a-875187f12659" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:41:13.036585 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.036534    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc1bee4-4948-4f8b-905a-875187f12659-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3bc1bee4-4948-4f8b-905a-875187f12659" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:41:13.036708 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.036625    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-kube-api-access-5djp5" (OuterVolumeSpecName: "kube-api-access-5djp5") pod "3bc1bee4-4948-4f8b-905a-875187f12659" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659"). InnerVolumeSpecName "kube-api-access-5djp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:41:13.036776 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.036702    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3bc1bee4-4948-4f8b-905a-875187f12659" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:41:13.036776 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.036740    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc1bee4-4948-4f8b-905a-875187f12659-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3bc1bee4-4948-4f8b-905a-875187f12659" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:41:13.042624 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.042599    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc1bee4-4948-4f8b-905a-875187f12659-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3bc1bee4-4948-4f8b-905a-875187f12659" (UID: "3bc1bee4-4948-4f8b-905a-875187f12659"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:41:13.063314 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.063272    2564 generic.go:356] "Generic (PLEG): container finished" podID="3bc1bee4-4948-4f8b-905a-875187f12659" containerID="4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f" exitCode=0
Jul 02 12:41:13.063415 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.063332    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-df765d5c7-sj6rs"
Jul 02 12:41:13.063415 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.063359    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-df765d5c7-sj6rs" event={"ID":"3bc1bee4-4948-4f8b-905a-875187f12659","Type":"ContainerDied","Data":"4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f"}
Jul 02 12:41:13.063498 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.063416    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-df765d5c7-sj6rs" event={"ID":"3bc1bee4-4948-4f8b-905a-875187f12659","Type":"ContainerDied","Data":"36520e3035ff8785f385aba6e90042a4cfda128de4979fd260412ff5ce6f6884"}
Jul 02 12:41:13.063498 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.063438    2564 scope.go:117] "RemoveContainer" containerID="4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f"
Jul 02 12:41:13.071888 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.071867    2564 scope.go:117] "RemoveContainer" containerID="4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f"
Jul 02 12:41:13.072115 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:41:13.072095    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f\": container with ID starting with 4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f not found: ID does not exist" containerID="4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f"
Jul 02 12:41:13.072186 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.072127    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f"} err="failed to get container status \"4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f\": rpc error: code = NotFound desc = could not find container \"4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f\": container with ID starting with 4f5b0d4d1195cdee47557a351df4167c9ffcfc043cee739585e25d217283db5f not found: ID does not exist"
Jul 02 12:41:13.088818 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.088791    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-df765d5c7-sj6rs"]
Jul 02 12:41:13.093665 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.093644    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-df765d5c7-sj6rs"]
Jul 02 12:41:13.136174 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.136148    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bc1bee4-4948-4f8b-905a-875187f12659-image-registry-private-configuration\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:13.136174 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.136173    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-bound-sa-token\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:13.136344 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.136183    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bc1bee4-4948-4f8b-905a-875187f12659-installation-pull-secrets\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:13.136344 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.136194    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5djp5\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-kube-api-access-5djp5\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:13.136344 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.136202    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bc1bee4-4948-4f8b-905a-875187f12659-ca-trust-extracted\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:13.136344 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.136211    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bc1bee4-4948-4f8b-905a-875187f12659-registry-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:13.136344 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:13.136219    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bc1bee4-4948-4f8b-905a-875187f12659-trusted-ca\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:14.422748 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:14.422712    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bc1bee4-4948-4f8b-905a-875187f12659" path="/var/lib/kubelet/pods/3bc1bee4-4948-4f8b-905a-875187f12659/volumes"
Jul 02 12:41:17.040235 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.040148    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-79695b9df5-zhmzj" podUID="f392a7a7-9314-471e-8c66-05658289559e" containerName="console" containerID="cri-o://505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54" gracePeriod=15
Jul 02 12:41:17.079816 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.079790    2564 generic.go:356] "Generic (PLEG): container finished" podID="49a6bea0-1155-4bd8-b8f6-f6b5a5016167" containerID="27d9a7424740a3ba09a511076c4dffad0c1e26d14865a90b39c9902044a3c746" exitCode=0
Jul 02 12:41:17.079933 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.079860    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-v24km" event={"ID":"49a6bea0-1155-4bd8-b8f6-f6b5a5016167","Type":"ContainerDied","Data":"27d9a7424740a3ba09a511076c4dffad0c1e26d14865a90b39c9902044a3c746"}
Jul 02 12:41:17.080187 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.080168    2564 scope.go:117] "RemoveContainer" containerID="27d9a7424740a3ba09a511076c4dffad0c1e26d14865a90b39c9902044a3c746"
Jul 02 12:41:17.275670 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.275648    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:41:17.366510 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.366437    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f392a7a7-9314-471e-8c66-05658289559e-console-serving-cert\") pod \"f392a7a7-9314-471e-8c66-05658289559e\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") "
Jul 02 12:41:17.366510 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.366497    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-oauth-serving-cert\") pod \"f392a7a7-9314-471e-8c66-05658289559e\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") "
Jul 02 12:41:17.366709 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.366552    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-console-config\") pod \"f392a7a7-9314-471e-8c66-05658289559e\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") "
Jul 02 12:41:17.366709 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.366585    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f392a7a7-9314-471e-8c66-05658289559e-console-oauth-config\") pod \"f392a7a7-9314-471e-8c66-05658289559e\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") "
Jul 02 12:41:17.366709 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.366622    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-service-ca\") pod \"f392a7a7-9314-471e-8c66-05658289559e\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") "
Jul 02 12:41:17.366709 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.366647    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lftfs\" (UniqueName: \"kubernetes.io/projected/f392a7a7-9314-471e-8c66-05658289559e-kube-api-access-lftfs\") pod \"f392a7a7-9314-471e-8c66-05658289559e\" (UID: \"f392a7a7-9314-471e-8c66-05658289559e\") "
Jul 02 12:41:17.366964 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.366922    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f392a7a7-9314-471e-8c66-05658289559e" (UID: "f392a7a7-9314-471e-8c66-05658289559e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:41:17.366964 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.366951    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-console-config" (OuterVolumeSpecName: "console-config") pod "f392a7a7-9314-471e-8c66-05658289559e" (UID: "f392a7a7-9314-471e-8c66-05658289559e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:41:17.367118 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.366957    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-service-ca" (OuterVolumeSpecName: "service-ca") pod "f392a7a7-9314-471e-8c66-05658289559e" (UID: "f392a7a7-9314-471e-8c66-05658289559e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:41:17.368790 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.368769    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f392a7a7-9314-471e-8c66-05658289559e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f392a7a7-9314-471e-8c66-05658289559e" (UID: "f392a7a7-9314-471e-8c66-05658289559e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:41:17.368868 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.368786    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f392a7a7-9314-471e-8c66-05658289559e-kube-api-access-lftfs" (OuterVolumeSpecName: "kube-api-access-lftfs") pod "f392a7a7-9314-471e-8c66-05658289559e" (UID: "f392a7a7-9314-471e-8c66-05658289559e"). InnerVolumeSpecName "kube-api-access-lftfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:41:17.368907 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.368889    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f392a7a7-9314-471e-8c66-05658289559e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f392a7a7-9314-471e-8c66-05658289559e" (UID: "f392a7a7-9314-471e-8c66-05658289559e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:41:17.468125 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.468100    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-console-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:17.468125 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.468122    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f392a7a7-9314-471e-8c66-05658289559e-console-oauth-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:17.468270 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.468132    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-service-ca\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:17.468270 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.468143    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lftfs\" (UniqueName: \"kubernetes.io/projected/f392a7a7-9314-471e-8c66-05658289559e-kube-api-access-lftfs\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:17.468270 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.468153    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f392a7a7-9314-471e-8c66-05658289559e-console-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:17.468270 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:17.468162    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f392a7a7-9314-471e-8c66-05658289559e-oauth-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:18.085641 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:18.085554    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-v24km" event={"ID":"49a6bea0-1155-4bd8-b8f6-f6b5a5016167","Type":"ContainerStarted","Data":"695fec619238b000f8bb2f64c85d3cd61af90f6cf7bb485981ff61aed1cb3be2"}
Jul 02 12:41:18.086848 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:18.086820    2564 generic.go:356] "Generic (PLEG): container finished" podID="f392a7a7-9314-471e-8c66-05658289559e" containerID="505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54" exitCode=0
Jul 02 12:41:18.086959 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:18.086864    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79695b9df5-zhmzj"
Jul 02 12:41:18.086959 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:18.086874    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79695b9df5-zhmzj" event={"ID":"f392a7a7-9314-471e-8c66-05658289559e","Type":"ContainerDied","Data":"505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54"}
Jul 02 12:41:18.086959 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:18.086915    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79695b9df5-zhmzj" event={"ID":"f392a7a7-9314-471e-8c66-05658289559e","Type":"ContainerDied","Data":"7ac85ea8c5e265c1fe351b34069dcd828f99cd97c7cba92937c05136bad6dd4d"}
Jul 02 12:41:18.086959 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:18.086941    2564 scope.go:117] "RemoveContainer" containerID="505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54"
Jul 02 12:41:18.095042 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:18.094995    2564 scope.go:117] "RemoveContainer" containerID="505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54"
Jul 02 12:41:18.095291 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:41:18.095272    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54\": container with ID starting with 505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54 not found: ID does not exist" containerID="505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54"
Jul 02 12:41:18.095379 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:18.095319    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54"} err="failed to get container status \"505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54\": rpc error: code = NotFound desc = could not find container \"505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54\": container with ID starting with 505e3bd7f47a71afb35aa41c129b612e31f6c4442927f8d11bbba5189621bd54 not found: ID does not exist"
Jul 02 12:41:18.126712 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:18.126684    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79695b9df5-zhmzj"]
Jul 02 12:41:18.130817 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:18.130788    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79695b9df5-zhmzj"]
Jul 02 12:41:18.421642 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:18.421566    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f392a7a7-9314-471e-8c66-05658289559e" path="/var/lib/kubelet/pods/f392a7a7-9314-471e-8c66-05658289559e/volumes"
Jul 02 12:41:27.121875 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:27.121840    2564 generic.go:356] "Generic (PLEG): container finished" podID="a15051df-16fa-46c6-8271-9351e5c7e12c" containerID="d9532706871e5885bddf39aa9b4faef96226476d1ac063367e740e12027400a7" exitCode=0
Jul 02 12:41:27.122233 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:27.121917    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf" event={"ID":"a15051df-16fa-46c6-8271-9351e5c7e12c","Type":"ContainerDied","Data":"d9532706871e5885bddf39aa9b4faef96226476d1ac063367e740e12027400a7"}
Jul 02 12:41:27.122275 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:27.122241    2564 scope.go:117] "RemoveContainer" containerID="d9532706871e5885bddf39aa9b4faef96226476d1ac063367e740e12027400a7"
Jul 02 12:41:28.127588 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:28.127551    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kjpgf" event={"ID":"a15051df-16fa-46c6-8271-9351e5c7e12c","Type":"ContainerStarted","Data":"4c18e917adcedf6f21c9d9a5b0c9932a61ed51ab85cbf1855a5ff4583a33886e"}
Jul 02 12:41:34.122383 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.122326    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7f698b95d6-j9vvs" podUID="9b5fec4e-e798-4944-9cbe-0197acb12b32" containerName="console" containerID="cri-o://18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7" gracePeriod=15
Jul 02 12:41:34.383780 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.383719    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:41:34.515209 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.515170    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-trusted-ca-bundle\") pod \"9b5fec4e-e798-4944-9cbe-0197acb12b32\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") "
Jul 02 12:41:34.515403 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.515220    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-config\") pod \"9b5fec4e-e798-4944-9cbe-0197acb12b32\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") "
Jul 02 12:41:34.515403 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.515243    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-oauth-config\") pod \"9b5fec4e-e798-4944-9cbe-0197acb12b32\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") "
Jul 02 12:41:34.515403 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.515288    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-service-ca\") pod \"9b5fec4e-e798-4944-9cbe-0197acb12b32\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") "
Jul 02 12:41:34.515403 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.515320    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-serving-cert\") pod \"9b5fec4e-e798-4944-9cbe-0197acb12b32\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") "
Jul 02 12:41:34.515403 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.515360    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgxtd\" (UniqueName: \"kubernetes.io/projected/9b5fec4e-e798-4944-9cbe-0197acb12b32-kube-api-access-dgxtd\") pod \"9b5fec4e-e798-4944-9cbe-0197acb12b32\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") "
Jul 02 12:41:34.515403 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.515378    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-oauth-serving-cert\") pod \"9b5fec4e-e798-4944-9cbe-0197acb12b32\" (UID: \"9b5fec4e-e798-4944-9cbe-0197acb12b32\") "
Jul 02 12:41:34.515700 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.515610    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9b5fec4e-e798-4944-9cbe-0197acb12b32" (UID: "9b5fec4e-e798-4944-9cbe-0197acb12b32"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:41:34.515877 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.515849    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-service-ca" (OuterVolumeSpecName: "service-ca") pod "9b5fec4e-e798-4944-9cbe-0197acb12b32" (UID: "9b5fec4e-e798-4944-9cbe-0197acb12b32"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:41:34.515999 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.515959    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9b5fec4e-e798-4944-9cbe-0197acb12b32" (UID: "9b5fec4e-e798-4944-9cbe-0197acb12b32"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:41:34.515999 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.515965    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-config" (OuterVolumeSpecName: "console-config") pod "9b5fec4e-e798-4944-9cbe-0197acb12b32" (UID: "9b5fec4e-e798-4944-9cbe-0197acb12b32"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:41:34.517577 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.517552    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9b5fec4e-e798-4944-9cbe-0197acb12b32" (UID: "9b5fec4e-e798-4944-9cbe-0197acb12b32"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:41:34.517677 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.517610    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9b5fec4e-e798-4944-9cbe-0197acb12b32" (UID: "9b5fec4e-e798-4944-9cbe-0197acb12b32"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:41:34.517677 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.517615    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5fec4e-e798-4944-9cbe-0197acb12b32-kube-api-access-dgxtd" (OuterVolumeSpecName: "kube-api-access-dgxtd") pod "9b5fec4e-e798-4944-9cbe-0197acb12b32" (UID: "9b5fec4e-e798-4944-9cbe-0197acb12b32"). InnerVolumeSpecName "kube-api-access-dgxtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:41:34.615885 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.615851    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-service-ca\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:34.615885 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.615879    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:34.615885 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.615889    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgxtd\" (UniqueName: \"kubernetes.io/projected/9b5fec4e-e798-4944-9cbe-0197acb12b32-kube-api-access-dgxtd\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:34.616094 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.615899    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-oauth-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:34.616094 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.615907    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-trusted-ca-bundle\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:34.616094 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.615915    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:34.616094 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:34.615923    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b5fec4e-e798-4944-9cbe-0197acb12b32-console-oauth-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:41:35.154568 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:35.154528    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b5fec4e-e798-4944-9cbe-0197acb12b32" containerID="18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7" exitCode=0
Jul 02 12:41:35.154965 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:35.154592    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f698b95d6-j9vvs"
Jul 02 12:41:35.154965 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:35.154617    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f698b95d6-j9vvs" event={"ID":"9b5fec4e-e798-4944-9cbe-0197acb12b32","Type":"ContainerDied","Data":"18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7"}
Jul 02 12:41:35.154965 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:35.154701    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f698b95d6-j9vvs" event={"ID":"9b5fec4e-e798-4944-9cbe-0197acb12b32","Type":"ContainerDied","Data":"a7f317929a1f936a347dd6b517a5055f68ca20a9f06d8a434c28d698884abcef"}
Jul 02 12:41:35.154965 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:35.154722    2564 scope.go:117] "RemoveContainer" containerID="18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7"
Jul 02 12:41:35.163044 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:35.163027    2564 scope.go:117] "RemoveContainer" containerID="18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7"
Jul 02 12:41:35.163337 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:41:35.163291    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7\": container with ID starting with 18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7 not found: ID does not exist" containerID="18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7"
Jul 02 12:41:35.163404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:35.163351    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7"} err="failed to get container status \"18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7\": rpc error: code = NotFound desc = could not find container \"18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7\": container with ID starting with 18285710c3cd54905af50c85bc0d3073f7bc5546993ae76147f241f137e750b7 not found: ID does not exist"
Jul 02 12:41:35.176471 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:35.176441    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f698b95d6-j9vvs"]
Jul 02 12:41:35.180792 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:35.180769    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f698b95d6-j9vvs"]
Jul 02 12:41:36.422313 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:41:36.422269    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b5fec4e-e798-4944-9cbe-0197acb12b32" path="/var/lib/kubelet/pods/9b5fec4e-e798-4944-9cbe-0197acb12b32/volumes"
Jul 02 12:42:16.277780 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.277746    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"]
Jul 02 12:42:16.278169 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.278007    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc1bee4-4948-4f8b-905a-875187f12659" containerName="registry"
Jul 02 12:42:16.278169 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.278019    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc1bee4-4948-4f8b-905a-875187f12659" containerName="registry"
Jul 02 12:42:16.278169 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.278036    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f392a7a7-9314-471e-8c66-05658289559e" containerName="console"
Jul 02 12:42:16.278169 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.278041    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f392a7a7-9314-471e-8c66-05658289559e" containerName="console"
Jul 02 12:42:16.278169 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.278049    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b5fec4e-e798-4944-9cbe-0197acb12b32" containerName="console"
Jul 02 12:42:16.278169 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.278055    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b5fec4e-e798-4944-9cbe-0197acb12b32" containerName="console"
Jul 02 12:42:16.278169 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.278097    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc1bee4-4948-4f8b-905a-875187f12659" containerName="registry"
Jul 02 12:42:16.278169 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.278104    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b5fec4e-e798-4944-9cbe-0197acb12b32" containerName="console"
Jul 02 12:42:16.278169 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.278110    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f392a7a7-9314-471e-8c66-05658289559e" containerName="console"
Jul 02 12:42:16.282499 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.282480    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.285329 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.285285    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 02 12:42:16.285670 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.285653    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 02 12:42:16.285763 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.285737    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 02 12:42:16.287290 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.287271    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 02 12:42:16.287400 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.287314    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 02 12:42:16.287646 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.287629    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bp6hw\""
Jul 02 12:42:16.293831 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.293810    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 02 12:42:16.304603 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.304581    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"]
Jul 02 12:42:16.316341 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.316321    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7982a147-27b6-4a68-bcd8-46df1943ba9f-serving-certs-ca-bundle\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.316443 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.316351    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7982a147-27b6-4a68-bcd8-46df1943ba9f-secret-telemeter-client\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.316443 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.316397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7982a147-27b6-4a68-bcd8-46df1943ba9f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.316443 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.316415    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7982a147-27b6-4a68-bcd8-46df1943ba9f-metrics-client-ca\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.316443 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.316431    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmkwk\" (UniqueName: \"kubernetes.io/projected/7982a147-27b6-4a68-bcd8-46df1943ba9f-kube-api-access-qmkwk\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.316583 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.316454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7982a147-27b6-4a68-bcd8-46df1943ba9f-federate-client-tls\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.316583 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.316542    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7982a147-27b6-4a68-bcd8-46df1943ba9f-telemeter-client-tls\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.316583 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.316573    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7982a147-27b6-4a68-bcd8-46df1943ba9f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.417101 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.417076    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7982a147-27b6-4a68-bcd8-46df1943ba9f-telemeter-client-tls\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.417288 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.417109    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7982a147-27b6-4a68-bcd8-46df1943ba9f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.417288 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.417154    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7982a147-27b6-4a68-bcd8-46df1943ba9f-serving-certs-ca-bundle\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.417288 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.417175    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7982a147-27b6-4a68-bcd8-46df1943ba9f-secret-telemeter-client\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.417288 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.417213    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7982a147-27b6-4a68-bcd8-46df1943ba9f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.417288 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.417239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7982a147-27b6-4a68-bcd8-46df1943ba9f-metrics-client-ca\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.417288 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.417264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkwk\" (UniqueName: \"kubernetes.io/projected/7982a147-27b6-4a68-bcd8-46df1943ba9f-kube-api-access-qmkwk\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.417288 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.417291    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7982a147-27b6-4a68-bcd8-46df1943ba9f-federate-client-tls\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.418554 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.418038    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7982a147-27b6-4a68-bcd8-46df1943ba9f-metrics-client-ca\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.418554 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.418276    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7982a147-27b6-4a68-bcd8-46df1943ba9f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.418812 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.418790    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7982a147-27b6-4a68-bcd8-46df1943ba9f-serving-certs-ca-bundle\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.419889 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.419853    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7982a147-27b6-4a68-bcd8-46df1943ba9f-federate-client-tls\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.420112 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.420092    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7982a147-27b6-4a68-bcd8-46df1943ba9f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.420198 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.420175    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7982a147-27b6-4a68-bcd8-46df1943ba9f-secret-telemeter-client\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.420235 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.420207    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7982a147-27b6-4a68-bcd8-46df1943ba9f-telemeter-client-tls\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.426859 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.426834    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkwk\" (UniqueName: \"kubernetes.io/projected/7982a147-27b6-4a68-bcd8-46df1943ba9f-kube-api-access-qmkwk\") pod \"telemeter-client-6cc4f64768-8d7dt\" (UID: \"7982a147-27b6-4a68-bcd8-46df1943ba9f\") " pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.592382 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.592286    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"
Jul 02 12:42:16.731813 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:16.731779    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cc4f64768-8d7dt"]
Jul 02 12:42:16.736348 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:42:16.736322    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7982a147_27b6_4a68_bcd8_46df1943ba9f.slice/crio-e284417d310eb2ea8d7d35c1847fd10331430488963f36393bb7c1a929ff35cc WatchSource:0}: Error finding container e284417d310eb2ea8d7d35c1847fd10331430488963f36393bb7c1a929ff35cc: Status 404 returned error can't find the container with id e284417d310eb2ea8d7d35c1847fd10331430488963f36393bb7c1a929ff35cc
Jul 02 12:42:17.305863 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:17.305822    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt" event={"ID":"7982a147-27b6-4a68-bcd8-46df1943ba9f","Type":"ContainerStarted","Data":"e284417d310eb2ea8d7d35c1847fd10331430488963f36393bb7c1a929ff35cc"}
Jul 02 12:42:19.316241 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:19.316204    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt" event={"ID":"7982a147-27b6-4a68-bcd8-46df1943ba9f","Type":"ContainerStarted","Data":"50b1bd53a3c0685b820257758b0bb8159428546fd444f0e25035047987b6e63a"}
Jul 02 12:42:19.316686 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:19.316260    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt" event={"ID":"7982a147-27b6-4a68-bcd8-46df1943ba9f","Type":"ContainerStarted","Data":"9fc46ee31bf61c1cfd503361a801e36158b9067b199b0af7d4014a197eaa4a20"}
Jul 02 12:42:19.316686 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:19.316280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt" event={"ID":"7982a147-27b6-4a68-bcd8-46df1943ba9f","Type":"ContainerStarted","Data":"ec9e7eb68af31a404e6dcee5bb03173ce1cb9f90fa479a8997c601c95adab0f7"}
Jul 02 12:42:19.339964 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:19.339914    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6cc4f64768-8d7dt" podStartSLOduration=1.582122869 podStartE2EDuration="3.339897641s" totalImagesPullingTime="1.757774772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:42:19.338485544 +0000 UTC m=+187.321959227" watchObservedRunningTime="2026-07-02 12:42:19.339897641 +0000 UTC m=+187.323371346"
Jul 02 12:42:20.235083 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.235047    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f7dfc959f-54zhs"]
Jul 02 12:42:20.238122 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.238108    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.250274 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.250235    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f7dfc959f-54zhs"]
Jul 02 12:42:20.349651 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.349624    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-console-config\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.349651 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.349653    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-service-ca\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.350011 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.349703    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3161b78b-dd52-499a-a11d-0857d55049b7-console-oauth-config\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.350011 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.349723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-oauth-serving-cert\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.350011 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.349742    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-trusted-ca-bundle\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.350011 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.349771    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cghdm\" (UniqueName: \"kubernetes.io/projected/3161b78b-dd52-499a-a11d-0857d55049b7-kube-api-access-cghdm\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.350011 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.349821    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3161b78b-dd52-499a-a11d-0857d55049b7-console-serving-cert\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.450203 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.450155    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-trusted-ca-bundle\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.450372 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.450341    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cghdm\" (UniqueName: \"kubernetes.io/projected/3161b78b-dd52-499a-a11d-0857d55049b7-kube-api-access-cghdm\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.450440 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.450408    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3161b78b-dd52-499a-a11d-0857d55049b7-console-serving-cert\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.450492 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.450440    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-console-config\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.450492 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.450468    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-service-ca\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.450767 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.450739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3161b78b-dd52-499a-a11d-0857d55049b7-console-oauth-config\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.450861 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.450786    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-oauth-serving-cert\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.451139 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.451117    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-console-config\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.451312 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.451202    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-trusted-ca-bundle\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.451405 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.451361    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-oauth-serving-cert\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.451569 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.451546    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-service-ca\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.453085 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.453063    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3161b78b-dd52-499a-a11d-0857d55049b7-console-oauth-config\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.453164 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.453139    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3161b78b-dd52-499a-a11d-0857d55049b7-console-serving-cert\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.458581 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.458563    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cghdm\" (UniqueName: \"kubernetes.io/projected/3161b78b-dd52-499a-a11d-0857d55049b7-kube-api-access-cghdm\") pod \"console-6f7dfc959f-54zhs\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") " pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.546745 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.546693    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:20.675818 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:20.675793    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f7dfc959f-54zhs"]
Jul 02 12:42:20.678382 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:42:20.678355    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3161b78b_dd52_499a_a11d_0857d55049b7.slice/crio-8cb35d2642b587597a229ace4cd4debd2ab18abe0a86974f959831d0664e82cc WatchSource:0}: Error finding container 8cb35d2642b587597a229ace4cd4debd2ab18abe0a86974f959831d0664e82cc: Status 404 returned error can't find the container with id 8cb35d2642b587597a229ace4cd4debd2ab18abe0a86974f959831d0664e82cc
Jul 02 12:42:21.325724 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:21.325685    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7dfc959f-54zhs" event={"ID":"3161b78b-dd52-499a-a11d-0857d55049b7","Type":"ContainerStarted","Data":"ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898"}
Jul 02 12:42:21.325925 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:21.325736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7dfc959f-54zhs" event={"ID":"3161b78b-dd52-499a-a11d-0857d55049b7","Type":"ContainerStarted","Data":"8cb35d2642b587597a229ace4cd4debd2ab18abe0a86974f959831d0664e82cc"}
Jul 02 12:42:21.345596 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:21.345546    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f7dfc959f-54zhs" podStartSLOduration=1.345531495 podStartE2EDuration="1.345531495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:42:21.344603791 +0000 UTC m=+189.328077473" watchObservedRunningTime="2026-07-02 12:42:21.345531495 +0000 UTC m=+189.329005177"
Jul 02 12:42:26.807757 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:26.807721    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f7dfc959f-54zhs"]
Jul 02 12:42:26.835175 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:26.835146    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8b7cbc7b7-wdtrg"]
Jul 02 12:42:26.838644 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:26.838625    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:26.851113 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:26.851090    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b7cbc7b7-wdtrg"]
Jul 02 12:42:26.905993 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:26.905965    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-config\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:26.906123 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:26.906005    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4b2r\" (UniqueName: \"kubernetes.io/projected/0921b634-18dd-43b4-90b2-234a7d7a5a9b-kube-api-access-z4b2r\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:26.906123 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:26.906071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-oauth-serving-cert\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:26.906123 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:26.906106    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-serving-cert\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:26.906256 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:26.906133    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-trusted-ca-bundle\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:26.906256 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:26.906162    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-service-ca\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:26.906256 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:26.906242    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-oauth-config\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.007184 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.007152    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4b2r\" (UniqueName: \"kubernetes.io/projected/0921b634-18dd-43b4-90b2-234a7d7a5a9b-kube-api-access-z4b2r\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.007354 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.007211    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-oauth-serving-cert\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.007354 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.007238    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-serving-cert\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.007354 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.007268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-trusted-ca-bundle\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.007354 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.007332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-service-ca\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.007562 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.007383    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-oauth-config\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.007562 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.007413    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-config\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.008113 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.008085    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-oauth-serving-cert\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.008113 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.008095    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-service-ca\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.008259 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.008164    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-trusted-ca-bundle\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.008259 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.008181    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-config\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.010040 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.010017    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-serving-cert\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.010154 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.010038    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-oauth-config\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.015504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.015480    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4b2r\" (UniqueName: \"kubernetes.io/projected/0921b634-18dd-43b4-90b2-234a7d7a5a9b-kube-api-access-z4b2r\") pod \"console-8b7cbc7b7-wdtrg\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") " pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.147601 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.147518    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:27.279458 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.279348    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b7cbc7b7-wdtrg"]
Jul 02 12:42:27.282341 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:42:27.282315    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0921b634_18dd_43b4_90b2_234a7d7a5a9b.slice/crio-eb11c60eae48c5020bf1a90e2dcde9fb7642bc01e4c275ef658e72887caabb00 WatchSource:0}: Error finding container eb11c60eae48c5020bf1a90e2dcde9fb7642bc01e4c275ef658e72887caabb00: Status 404 returned error can't find the container with id eb11c60eae48c5020bf1a90e2dcde9fb7642bc01e4c275ef658e72887caabb00
Jul 02 12:42:27.348810 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.348780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b7cbc7b7-wdtrg" event={"ID":"0921b634-18dd-43b4-90b2-234a7d7a5a9b","Type":"ContainerStarted","Data":"89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc"}
Jul 02 12:42:27.348912 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.348829    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b7cbc7b7-wdtrg" event={"ID":"0921b634-18dd-43b4-90b2-234a7d7a5a9b","Type":"ContainerStarted","Data":"eb11c60eae48c5020bf1a90e2dcde9fb7642bc01e4c275ef658e72887caabb00"}
Jul 02 12:42:27.368054 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:27.368010    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8b7cbc7b7-wdtrg" podStartSLOduration=1.367994278 podStartE2EDuration="1.367994278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:42:27.366758537 +0000 UTC m=+195.350232243" watchObservedRunningTime="2026-07-02 12:42:27.367994278 +0000 UTC m=+195.351467961"
Jul 02 12:42:30.547013 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:30.546974    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:37.147790 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:37.147749    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:37.148186 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:37.147817    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:37.152562 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:37.152539    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:37.394265 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:37.394238    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:42:37.442694 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:37.442669    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77fc79b6fb-wk7w2"]
Jul 02 12:42:51.827310 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:51.827246    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6f7dfc959f-54zhs" podUID="3161b78b-dd52-499a-a11d-0857d55049b7" containerName="console" containerID="cri-o://ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898" gracePeriod=15
Jul 02 12:42:52.072709 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.072686    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:52.112091 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.112023    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-trusted-ca-bundle\") pod \"3161b78b-dd52-499a-a11d-0857d55049b7\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") "
Jul 02 12:42:52.112091 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.112069    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3161b78b-dd52-499a-a11d-0857d55049b7-console-serving-cert\") pod \"3161b78b-dd52-499a-a11d-0857d55049b7\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") "
Jul 02 12:42:52.112287 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.112117    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cghdm\" (UniqueName: \"kubernetes.io/projected/3161b78b-dd52-499a-a11d-0857d55049b7-kube-api-access-cghdm\") pod \"3161b78b-dd52-499a-a11d-0857d55049b7\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") "
Jul 02 12:42:52.112287 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.112162    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3161b78b-dd52-499a-a11d-0857d55049b7-console-oauth-config\") pod \"3161b78b-dd52-499a-a11d-0857d55049b7\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") "
Jul 02 12:42:52.112287 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.112180    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-service-ca\") pod \"3161b78b-dd52-499a-a11d-0857d55049b7\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") "
Jul 02 12:42:52.112287 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.112214    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-console-config\") pod \"3161b78b-dd52-499a-a11d-0857d55049b7\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") "
Jul 02 12:42:52.112287 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.112239    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-oauth-serving-cert\") pod \"3161b78b-dd52-499a-a11d-0857d55049b7\" (UID: \"3161b78b-dd52-499a-a11d-0857d55049b7\") "
Jul 02 12:42:52.112570 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.112500    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3161b78b-dd52-499a-a11d-0857d55049b7" (UID: "3161b78b-dd52-499a-a11d-0857d55049b7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:42:52.112777 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.112753    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3161b78b-dd52-499a-a11d-0857d55049b7" (UID: "3161b78b-dd52-499a-a11d-0857d55049b7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:42:52.113379 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.113334    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-service-ca" (OuterVolumeSpecName: "service-ca") pod "3161b78b-dd52-499a-a11d-0857d55049b7" (UID: "3161b78b-dd52-499a-a11d-0857d55049b7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:42:52.113468 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.113404    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-console-config" (OuterVolumeSpecName: "console-config") pod "3161b78b-dd52-499a-a11d-0857d55049b7" (UID: "3161b78b-dd52-499a-a11d-0857d55049b7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:42:52.114350 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.114324    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3161b78b-dd52-499a-a11d-0857d55049b7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3161b78b-dd52-499a-a11d-0857d55049b7" (UID: "3161b78b-dd52-499a-a11d-0857d55049b7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:42:52.114531 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.114506    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3161b78b-dd52-499a-a11d-0857d55049b7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3161b78b-dd52-499a-a11d-0857d55049b7" (UID: "3161b78b-dd52-499a-a11d-0857d55049b7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:42:52.114622 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.114515    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3161b78b-dd52-499a-a11d-0857d55049b7-kube-api-access-cghdm" (OuterVolumeSpecName: "kube-api-access-cghdm") pod "3161b78b-dd52-499a-a11d-0857d55049b7" (UID: "3161b78b-dd52-499a-a11d-0857d55049b7"). InnerVolumeSpecName "kube-api-access-cghdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:42:52.213071 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.213038    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3161b78b-dd52-499a-a11d-0857d55049b7-console-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:42:52.213071 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.213066    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cghdm\" (UniqueName: \"kubernetes.io/projected/3161b78b-dd52-499a-a11d-0857d55049b7-kube-api-access-cghdm\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:42:52.213071 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.213076    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3161b78b-dd52-499a-a11d-0857d55049b7-console-oauth-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:42:52.213277 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.213085    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-service-ca\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:42:52.213277 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.213094    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-console-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:42:52.213277 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.213112    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-oauth-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:42:52.213277 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.213121    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3161b78b-dd52-499a-a11d-0857d55049b7-trusted-ca-bundle\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:42:52.448888 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.448853    2564 generic.go:356] "Generic (PLEG): container finished" podID="3161b78b-dd52-499a-a11d-0857d55049b7" containerID="ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898" exitCode=0
Jul 02 12:42:52.449000 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.448904    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7dfc959f-54zhs"
Jul 02 12:42:52.449000 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.448918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7dfc959f-54zhs" event={"ID":"3161b78b-dd52-499a-a11d-0857d55049b7","Type":"ContainerDied","Data":"ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898"}
Jul 02 12:42:52.449000 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.448961    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7dfc959f-54zhs" event={"ID":"3161b78b-dd52-499a-a11d-0857d55049b7","Type":"ContainerDied","Data":"8cb35d2642b587597a229ace4cd4debd2ab18abe0a86974f959831d0664e82cc"}
Jul 02 12:42:52.449000 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.448981    2564 scope.go:117] "RemoveContainer" containerID="ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898"
Jul 02 12:42:52.456749 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.456731    2564 scope.go:117] "RemoveContainer" containerID="ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898"
Jul 02 12:42:52.456979 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:42:52.456961    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898\": container with ID starting with ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898 not found: ID does not exist" containerID="ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898"
Jul 02 12:42:52.457057 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.456985    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898"} err="failed to get container status \"ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898\": rpc error: code = NotFound desc = could not find container \"ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898\": container with ID starting with ce3604dd6612d943620f99dd5e148849dc929ed503a3b49518dd5888f80a3898 not found: ID does not exist"
Jul 02 12:42:52.465923 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.465901    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f7dfc959f-54zhs"]
Jul 02 12:42:52.469728 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:52.469709    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f7dfc959f-54zhs"]
Jul 02 12:42:54.422060 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:42:54.422031    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3161b78b-dd52-499a-a11d-0857d55049b7" path="/var/lib/kubelet/pods/3161b78b-dd52-499a-a11d-0857d55049b7/volumes"
Jul 02 12:43:02.462747 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.462703    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-77fc79b6fb-wk7w2" podUID="92fd9b58-5e07-4999-9762-7c329becd682" containerName="console" containerID="cri-o://8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136" gracePeriod=15
Jul 02 12:43:02.692187 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.692165    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:43:02.794676 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.794598    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92fd9b58-5e07-4999-9762-7c329becd682-console-serving-cert\") pod \"92fd9b58-5e07-4999-9762-7c329becd682\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") "
Jul 02 12:43:02.794676 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.794639    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5th6r\" (UniqueName: \"kubernetes.io/projected/92fd9b58-5e07-4999-9762-7c329becd682-kube-api-access-5th6r\") pod \"92fd9b58-5e07-4999-9762-7c329becd682\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") "
Jul 02 12:43:02.794676 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.794674    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-console-config\") pod \"92fd9b58-5e07-4999-9762-7c329becd682\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") "
Jul 02 12:43:02.794931 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.794718    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92fd9b58-5e07-4999-9762-7c329becd682-console-oauth-config\") pod \"92fd9b58-5e07-4999-9762-7c329becd682\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") "
Jul 02 12:43:02.794931 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.794742    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-trusted-ca-bundle\") pod \"92fd9b58-5e07-4999-9762-7c329becd682\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") "
Jul 02 12:43:02.794931 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.794764    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-service-ca\") pod \"92fd9b58-5e07-4999-9762-7c329becd682\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") "
Jul 02 12:43:02.794931 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.794805    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-oauth-serving-cert\") pod \"92fd9b58-5e07-4999-9762-7c329becd682\" (UID: \"92fd9b58-5e07-4999-9762-7c329becd682\") "
Jul 02 12:43:02.795234 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.795124    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-console-config" (OuterVolumeSpecName: "console-config") pod "92fd9b58-5e07-4999-9762-7c329becd682" (UID: "92fd9b58-5e07-4999-9762-7c329becd682"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:43:02.795390 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.795213    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "92fd9b58-5e07-4999-9762-7c329becd682" (UID: "92fd9b58-5e07-4999-9762-7c329becd682"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:43:02.795390 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.795282    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-service-ca" (OuterVolumeSpecName: "service-ca") pod "92fd9b58-5e07-4999-9762-7c329becd682" (UID: "92fd9b58-5e07-4999-9762-7c329becd682"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:43:02.795390 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.795326    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "92fd9b58-5e07-4999-9762-7c329becd682" (UID: "92fd9b58-5e07-4999-9762-7c329becd682"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:43:02.796901 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.796867    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fd9b58-5e07-4999-9762-7c329becd682-kube-api-access-5th6r" (OuterVolumeSpecName: "kube-api-access-5th6r") pod "92fd9b58-5e07-4999-9762-7c329becd682" (UID: "92fd9b58-5e07-4999-9762-7c329becd682"). InnerVolumeSpecName "kube-api-access-5th6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:43:02.796992 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.796933    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fd9b58-5e07-4999-9762-7c329becd682-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "92fd9b58-5e07-4999-9762-7c329becd682" (UID: "92fd9b58-5e07-4999-9762-7c329becd682"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:43:02.796992 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.796945    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fd9b58-5e07-4999-9762-7c329becd682-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "92fd9b58-5e07-4999-9762-7c329becd682" (UID: "92fd9b58-5e07-4999-9762-7c329becd682"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:43:02.896270 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.896244    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92fd9b58-5e07-4999-9762-7c329becd682-console-oauth-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:43:02.896270 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.896267    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-trusted-ca-bundle\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:43:02.896434 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.896279    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-service-ca\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:43:02.896434 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.896288    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-oauth-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:43:02.896434 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.896308    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92fd9b58-5e07-4999-9762-7c329becd682-console-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:43:02.896434 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.896317    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5th6r\" (UniqueName: \"kubernetes.io/projected/92fd9b58-5e07-4999-9762-7c329becd682-kube-api-access-5th6r\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:43:02.896434 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:02.896326    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92fd9b58-5e07-4999-9762-7c329becd682-console-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:43:03.489924 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:03.489890    2564 generic.go:356] "Generic (PLEG): container finished" podID="92fd9b58-5e07-4999-9762-7c329becd682" containerID="8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136" exitCode=0
Jul 02 12:43:03.490410 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:03.489939    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77fc79b6fb-wk7w2"
Jul 02 12:43:03.490410 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:03.489985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77fc79b6fb-wk7w2" event={"ID":"92fd9b58-5e07-4999-9762-7c329becd682","Type":"ContainerDied","Data":"8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136"}
Jul 02 12:43:03.490410 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:03.490046    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77fc79b6fb-wk7w2" event={"ID":"92fd9b58-5e07-4999-9762-7c329becd682","Type":"ContainerDied","Data":"d0bd6be30b70413952d3e0c2c5281708ed229b0a4666f6250fa74aa0b3d8b0df"}
Jul 02 12:43:03.490410 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:03.490081    2564 scope.go:117] "RemoveContainer" containerID="8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136"
Jul 02 12:43:03.498209 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:03.498192    2564 scope.go:117] "RemoveContainer" containerID="8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136"
Jul 02 12:43:03.498528 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:43:03.498504    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136\": container with ID starting with 8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136 not found: ID does not exist" containerID="8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136"
Jul 02 12:43:03.498641 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:03.498533    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136"} err="failed to get container status \"8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136\": rpc error: code = NotFound desc = could not find container \"8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136\": container with ID starting with 8e58052204015fc595befcc25ba1b7b71f25bf15736d622795b434c1fb4b5136 not found: ID does not exist"
Jul 02 12:43:03.512625 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:03.512595    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77fc79b6fb-wk7w2"]
Jul 02 12:43:03.518404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:03.518385    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77fc79b6fb-wk7w2"]
Jul 02 12:43:04.421630 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:43:04.421602    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92fd9b58-5e07-4999-9762-7c329becd682" path="/var/lib/kubelet/pods/92fd9b58-5e07-4999-9762-7c329becd682/volumes"
Jul 02 12:44:12.366808 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:12.366771    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 12:44:42.421366 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.421336    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"]
Jul 02 12:44:42.423695 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.421570    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92fd9b58-5e07-4999-9762-7c329becd682" containerName="console"
Jul 02 12:44:42.423695 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.421579    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="92fd9b58-5e07-4999-9762-7c329becd682" containerName="console"
Jul 02 12:44:42.423695 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.421601    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3161b78b-dd52-499a-a11d-0857d55049b7" containerName="console"
Jul 02 12:44:42.423695 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.421606    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3161b78b-dd52-499a-a11d-0857d55049b7" containerName="console"
Jul 02 12:44:42.423695 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.421645    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="92fd9b58-5e07-4999-9762-7c329becd682" containerName="console"
Jul 02 12:44:42.423695 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.421654    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3161b78b-dd52-499a-a11d-0857d55049b7" containerName="console"
Jul 02 12:44:42.424607 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.424591    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:44:42.427269 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.427247    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 02 12:44:42.427401 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.427291    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 02 12:44:42.427401 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.427359    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l9ckv\""
Jul 02 12:44:42.440444 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.440419    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"]
Jul 02 12:44:42.567471 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.567435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87z8\" (UniqueName: \"kubernetes.io/projected/aa16b4be-50c5-4489-899c-b78c7bebcb8e-kube-api-access-b87z8\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b\" (UID: \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:44:42.567471 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.567477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa16b4be-50c5-4489-899c-b78c7bebcb8e-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b\" (UID: \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:44:42.567702 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.567544    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa16b4be-50c5-4489-899c-b78c7bebcb8e-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b\" (UID: \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:44:42.667973 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.667927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87z8\" (UniqueName: \"kubernetes.io/projected/aa16b4be-50c5-4489-899c-b78c7bebcb8e-kube-api-access-b87z8\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b\" (UID: \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:44:42.668156 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.667988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa16b4be-50c5-4489-899c-b78c7bebcb8e-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b\" (UID: \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:44:42.668156 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.668072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa16b4be-50c5-4489-899c-b78c7bebcb8e-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b\" (UID: \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:44:42.668427 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.668404    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa16b4be-50c5-4489-899c-b78c7bebcb8e-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b\" (UID: \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:44:42.668468 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.668416    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa16b4be-50c5-4489-899c-b78c7bebcb8e-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b\" (UID: \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:44:42.676330 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.676237    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87z8\" (UniqueName: \"kubernetes.io/projected/aa16b4be-50c5-4489-899c-b78c7bebcb8e-kube-api-access-b87z8\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b\" (UID: \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:44:42.733674 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.733647    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:44:42.859269 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.859235    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"]
Jul 02 12:44:42.862272 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:44:42.862249    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa16b4be_50c5_4489_899c_b78c7bebcb8e.slice/crio-0ca42a5e07451fefea1134f9f3c9988091adb5248f53646f3d4bbe08ad271273 WatchSource:0}: Error finding container 0ca42a5e07451fefea1134f9f3c9988091adb5248f53646f3d4bbe08ad271273: Status 404 returned error can't find the container with id 0ca42a5e07451fefea1134f9f3c9988091adb5248f53646f3d4bbe08ad271273
Jul 02 12:44:42.864139 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:42.864119    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 12:44:43.851354 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:43.851318    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b" event={"ID":"aa16b4be-50c5-4489-899c-b78c7bebcb8e","Type":"ContainerStarted","Data":"0ca42a5e07451fefea1134f9f3c9988091adb5248f53646f3d4bbe08ad271273"}
Jul 02 12:44:49.878380 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:49.878340    2564 generic.go:356] "Generic (PLEG): container finished" podID="aa16b4be-50c5-4489-899c-b78c7bebcb8e" containerID="c8baff55e43d41c41452f2022d27c50a4bba3419e2093f45634822f896237eb1" exitCode=0
Jul 02 12:44:49.878816 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:49.878418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b" event={"ID":"aa16b4be-50c5-4489-899c-b78c7bebcb8e","Type":"ContainerDied","Data":"c8baff55e43d41c41452f2022d27c50a4bba3419e2093f45634822f896237eb1"}
Jul 02 12:44:52.891782 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:52.891751    2564 generic.go:356] "Generic (PLEG): container finished" podID="aa16b4be-50c5-4489-899c-b78c7bebcb8e" containerID="6cd0304f41d1d839e742f3244574415d0d3e04405de360bbb6bbdad570d475cd" exitCode=0
Jul 02 12:44:52.892161 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:52.891812    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b" event={"ID":"aa16b4be-50c5-4489-899c-b78c7bebcb8e","Type":"ContainerDied","Data":"6cd0304f41d1d839e742f3244574415d0d3e04405de360bbb6bbdad570d475cd"}
Jul 02 12:44:58.918369 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:58.918336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b" event={"ID":"aa16b4be-50c5-4489-899c-b78c7bebcb8e","Type":"ContainerStarted","Data":"de371f8e228a7074e5de429544c021bb176f4b687e778807f6529b04575b7e02"}
Jul 02 12:44:58.939409 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:58.939357    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b" podStartSLOduration=2.315279469 podStartE2EDuration="16.939336746s" totalImagesPullingTime="14.624057277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:44:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:44:58.9374243 +0000 UTC m=+346.920897982" watchObservedRunningTime="2026-07-02 12:44:58.939336746 +0000 UTC m=+346.922810432"
Jul 02 12:44:59.924447 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:59.924414    2564 generic.go:356] "Generic (PLEG): container finished" podID="aa16b4be-50c5-4489-899c-b78c7bebcb8e" containerID="de371f8e228a7074e5de429544c021bb176f4b687e778807f6529b04575b7e02" exitCode=0
Jul 02 12:44:59.924839 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:44:59.924506    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b" event={"ID":"aa16b4be-50c5-4489-899c-b78c7bebcb8e","Type":"ContainerDied","Data":"de371f8e228a7074e5de429544c021bb176f4b687e778807f6529b04575b7e02"}
Jul 02 12:45:01.053207 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.053181    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:45:01.122194 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.122163    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa16b4be-50c5-4489-899c-b78c7bebcb8e-util\") pod \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\" (UID: \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\") "
Jul 02 12:45:01.122368 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.122207    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b87z8\" (UniqueName: \"kubernetes.io/projected/aa16b4be-50c5-4489-899c-b78c7bebcb8e-kube-api-access-b87z8\") pod \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\" (UID: \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\") "
Jul 02 12:45:01.122368 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.122230    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa16b4be-50c5-4489-899c-b78c7bebcb8e-bundle\") pod \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\" (UID: \"aa16b4be-50c5-4489-899c-b78c7bebcb8e\") "
Jul 02 12:45:01.123020 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.122985    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa16b4be-50c5-4489-899c-b78c7bebcb8e-bundle" (OuterVolumeSpecName: "bundle") pod "aa16b4be-50c5-4489-899c-b78c7bebcb8e" (UID: "aa16b4be-50c5-4489-899c-b78c7bebcb8e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:45:01.124288 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.124266    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa16b4be-50c5-4489-899c-b78c7bebcb8e-kube-api-access-b87z8" (OuterVolumeSpecName: "kube-api-access-b87z8") pod "aa16b4be-50c5-4489-899c-b78c7bebcb8e" (UID: "aa16b4be-50c5-4489-899c-b78c7bebcb8e"). InnerVolumeSpecName "kube-api-access-b87z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:45:01.127422 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.127398    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa16b4be-50c5-4489-899c-b78c7bebcb8e-util" (OuterVolumeSpecName: "util") pod "aa16b4be-50c5-4489-899c-b78c7bebcb8e" (UID: "aa16b4be-50c5-4489-899c-b78c7bebcb8e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:45:01.223264 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.223242    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa16b4be-50c5-4489-899c-b78c7bebcb8e-util\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:45:01.223264 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.223264    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b87z8\" (UniqueName: \"kubernetes.io/projected/aa16b4be-50c5-4489-899c-b78c7bebcb8e-kube-api-access-b87z8\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:45:01.223435 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.223275    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa16b4be-50c5-4489-899c-b78c7bebcb8e-bundle\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:45:01.933215 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.933178    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b" event={"ID":"aa16b4be-50c5-4489-899c-b78c7bebcb8e","Type":"ContainerDied","Data":"0ca42a5e07451fefea1134f9f3c9988091adb5248f53646f3d4bbe08ad271273"}
Jul 02 12:45:01.933215 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.933219    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5894vr5b"
Jul 02 12:45:01.933451 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:01.933225    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca42a5e07451fefea1134f9f3c9988091adb5248f53646f3d4bbe08ad271273"
Jul 02 12:45:04.632526 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.632493    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"]
Jul 02 12:45:04.632885 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.632787    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa16b4be-50c5-4489-899c-b78c7bebcb8e" containerName="util"
Jul 02 12:45:04.632885 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.632800    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa16b4be-50c5-4489-899c-b78c7bebcb8e" containerName="util"
Jul 02 12:45:04.632885 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.632810    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa16b4be-50c5-4489-899c-b78c7bebcb8e" containerName="pull"
Jul 02 12:45:04.632885 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.632815    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa16b4be-50c5-4489-899c-b78c7bebcb8e" containerName="pull"
Jul 02 12:45:04.632885 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.632826    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa16b4be-50c5-4489-899c-b78c7bebcb8e" containerName="extract"
Jul 02 12:45:04.632885 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.632833    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa16b4be-50c5-4489-899c-b78c7bebcb8e" containerName="extract"
Jul 02 12:45:04.633086 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.632898    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa16b4be-50c5-4489-899c-b78c7bebcb8e" containerName="extract"
Jul 02 12:45:04.665898 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.665864    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"]
Jul 02 12:45:04.666054 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.665983    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"
Jul 02 12:45:04.668819 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.668795    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 02 12:45:04.668975 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.668946    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-xczgt\""
Jul 02 12:45:04.669035 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.668948    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 02 12:45:04.677807 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.677788    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 02 12:45:04.749536 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.749496    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k\" (UID: \"5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"
Jul 02 12:45:04.749713 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.749603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q29wp\" (UniqueName: \"kubernetes.io/projected/5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7-kube-api-access-q29wp\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k\" (UID: \"5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"
Jul 02 12:45:04.850122 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.850083    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k\" (UID: \"5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"
Jul 02 12:45:04.850327 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.850156    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q29wp\" (UniqueName: \"kubernetes.io/projected/5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7-kube-api-access-q29wp\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k\" (UID: \"5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"
Jul 02 12:45:04.852442 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.852419    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k\" (UID: \"5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"
Jul 02 12:45:04.858430 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.858404    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q29wp\" (UniqueName: \"kubernetes.io/projected/5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7-kube-api-access-q29wp\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k\" (UID: \"5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"
Jul 02 12:45:04.976169 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:04.976129    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"
Jul 02 12:45:05.131152 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:05.131127    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"]
Jul 02 12:45:05.134110 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:45:05.134083    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0ab77d_a03f_4bfb_9fe1_acdaf72973b7.slice/crio-45e568ee68dea27f9c457c478a87dbce4b342d4bd0e1aeea0705bfefe244fd48 WatchSource:0}: Error finding container 45e568ee68dea27f9c457c478a87dbce4b342d4bd0e1aeea0705bfefe244fd48: Status 404 returned error can't find the container with id 45e568ee68dea27f9c457c478a87dbce4b342d4bd0e1aeea0705bfefe244fd48
Jul 02 12:45:05.948824 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:05.948790    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k" event={"ID":"5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7","Type":"ContainerStarted","Data":"45e568ee68dea27f9c457c478a87dbce4b342d4bd0e1aeea0705bfefe244fd48"}
Jul 02 12:45:09.934347 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:09.934313    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-dtrt9"]
Jul 02 12:45:09.952047 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:09.952009    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-dtrt9"]
Jul 02 12:45:09.952207 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:09.952147    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:09.954885 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:09.954861    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 02 12:45:09.955018 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:09.954860    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 02 12:45:09.955018 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:09.954978    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-7l85w\""
Jul 02 12:45:09.967698 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:09.967677    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k" event={"ID":"5e0ab77d-a03f-4bfb-9fe1-acdaf72973b7","Type":"ContainerStarted","Data":"045464d9fd387a953aa166343291e47e4c51de0e9bc549aa7a4ad846bc74423e"}
Jul 02 12:45:09.967813 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:09.967728    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"
Jul 02 12:45:09.989710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:09.989661    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k" podStartSLOduration=2.106581701 podStartE2EDuration="5.989647161s" totalImagesPullingTime="3.88306546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:45:09.987929632 +0000 UTC m=+357.971403316" watchObservedRunningTime="2026-07-02 12:45:09.989647161 +0000 UTC m=+357.973120843"
Jul 02 12:45:09.990506 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:09.990483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkpv\" (UniqueName: \"kubernetes.io/projected/78e28ae0-8984-461c-bf51-c0283231dc13-kube-api-access-wrkpv\") pod \"keda-operator-68bff59c-dtrt9\" (UID: \"78e28ae0-8984-461c-bf51-c0283231dc13\") " pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:09.990645 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:09.990625    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/78e28ae0-8984-461c-bf51-c0283231dc13-certificates\") pod \"keda-operator-68bff59c-dtrt9\" (UID: \"78e28ae0-8984-461c-bf51-c0283231dc13\") " pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:09.990695 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:09.990668    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/78e28ae0-8984-461c-bf51-c0283231dc13-cabundle0\") pod \"keda-operator-68bff59c-dtrt9\" (UID: \"78e28ae0-8984-461c-bf51-c0283231dc13\") " pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:10.091287 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.091255    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkpv\" (UniqueName: \"kubernetes.io/projected/78e28ae0-8984-461c-bf51-c0283231dc13-kube-api-access-wrkpv\") pod \"keda-operator-68bff59c-dtrt9\" (UID: \"78e28ae0-8984-461c-bf51-c0283231dc13\") " pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:10.091479 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.091314    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/78e28ae0-8984-461c-bf51-c0283231dc13-certificates\") pod \"keda-operator-68bff59c-dtrt9\" (UID: \"78e28ae0-8984-461c-bf51-c0283231dc13\") " pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:10.091479 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.091340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/78e28ae0-8984-461c-bf51-c0283231dc13-cabundle0\") pod \"keda-operator-68bff59c-dtrt9\" (UID: \"78e28ae0-8984-461c-bf51-c0283231dc13\") " pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:10.091605 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:45:10.091498    2564 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 02 12:45:10.091605 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:45:10.091518    2564 secret.go:281] references non-existent secret key: ca.crt
Jul 02 12:45:10.091605 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:45:10.091528    2564 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 02 12:45:10.091605 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:45:10.091543    2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-dtrt9: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 02 12:45:10.091821 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:45:10.091619    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78e28ae0-8984-461c-bf51-c0283231dc13-certificates podName:78e28ae0-8984-461c-bf51-c0283231dc13 nodeName:}" failed. No retries permitted until 2026-07-02 12:45:10.59159497 +0000 UTC m=+358.575068645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/78e28ae0-8984-461c-bf51-c0283231dc13-certificates") pod "keda-operator-68bff59c-dtrt9" (UID: "78e28ae0-8984-461c-bf51-c0283231dc13") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 02 12:45:10.092123 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.092092    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/78e28ae0-8984-461c-bf51-c0283231dc13-cabundle0\") pod \"keda-operator-68bff59c-dtrt9\" (UID: \"78e28ae0-8984-461c-bf51-c0283231dc13\") " pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:10.104737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.104705    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkpv\" (UniqueName: \"kubernetes.io/projected/78e28ae0-8984-461c-bf51-c0283231dc13-kube-api-access-wrkpv\") pod \"keda-operator-68bff59c-dtrt9\" (UID: \"78e28ae0-8984-461c-bf51-c0283231dc13\") " pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:10.192757 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.192676    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"]
Jul 02 12:45:10.213230 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.213202    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"]
Jul 02 12:45:10.213406 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.213318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:10.216494 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.216472    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 02 12:45:10.293105 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.293065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945-certificates\") pod \"keda-metrics-apiserver-559f75f947-2mps9\" (UID: \"5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:10.293270 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.293149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-2mps9\" (UID: \"5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:10.293270 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.293186    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vj7z\" (UniqueName: \"kubernetes.io/projected/5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945-kube-api-access-9vj7z\") pod \"keda-metrics-apiserver-559f75f947-2mps9\" (UID: \"5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:10.394271 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.394235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-2mps9\" (UID: \"5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:10.394471 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.394280    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vj7z\" (UniqueName: \"kubernetes.io/projected/5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945-kube-api-access-9vj7z\") pod \"keda-metrics-apiserver-559f75f947-2mps9\" (UID: \"5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:10.394471 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.394441    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945-certificates\") pod \"keda-metrics-apiserver-559f75f947-2mps9\" (UID: \"5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:10.394606 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:45:10.394588    2564 secret.go:281] references non-existent secret key: tls.crt
Jul 02 12:45:10.394643 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:45:10.394613    2564 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 02 12:45:10.394643 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:45:10.394640    2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-2mps9: references non-existent secret key: tls.crt
Jul 02 12:45:10.394710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.394686    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-2mps9\" (UID: \"5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:10.394710 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:45:10.394704    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945-certificates podName:5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945 nodeName:}" failed. No retries permitted until 2026-07-02 12:45:10.894684987 +0000 UTC m=+358.878158665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945-certificates") pod "keda-metrics-apiserver-559f75f947-2mps9" (UID: "5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945") : references non-existent secret key: tls.crt
Jul 02 12:45:10.404091 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.404065    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vj7z\" (UniqueName: \"kubernetes.io/projected/5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945-kube-api-access-9vj7z\") pod \"keda-metrics-apiserver-559f75f947-2mps9\" (UID: \"5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:10.596452 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.596363    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/78e28ae0-8984-461c-bf51-c0283231dc13-certificates\") pod \"keda-operator-68bff59c-dtrt9\" (UID: \"78e28ae0-8984-461c-bf51-c0283231dc13\") " pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:10.598725 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.598699    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/78e28ae0-8984-461c-bf51-c0283231dc13-certificates\") pod \"keda-operator-68bff59c-dtrt9\" (UID: \"78e28ae0-8984-461c-bf51-c0283231dc13\") " pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:10.709129 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.709092    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-m7mwg"]
Jul 02 12:45:10.730980 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.730951    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-m7mwg"]
Jul 02 12:45:10.731114 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.731062    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg"
Jul 02 12:45:10.733783 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.733761    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 02 12:45:10.797970 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.797935    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmjl\" (UniqueName: \"kubernetes.io/projected/e2e89261-d1af-4dd1-af81-203e880592ee-kube-api-access-nqmjl\") pod \"keda-admission-5fbd5c4644-m7mwg\" (UID: \"e2e89261-d1af-4dd1-af81-203e880592ee\") " pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg"
Jul 02 12:45:10.798132 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.798023    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e2e89261-d1af-4dd1-af81-203e880592ee-certificates\") pod \"keda-admission-5fbd5c4644-m7mwg\" (UID: \"e2e89261-d1af-4dd1-af81-203e880592ee\") " pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg"
Jul 02 12:45:10.862230 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.862152    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:10.898961 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.898931    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e2e89261-d1af-4dd1-af81-203e880592ee-certificates\") pod \"keda-admission-5fbd5c4644-m7mwg\" (UID: \"e2e89261-d1af-4dd1-af81-203e880592ee\") " pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg"
Jul 02 12:45:10.899105 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.899004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945-certificates\") pod \"keda-metrics-apiserver-559f75f947-2mps9\" (UID: \"5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:10.899105 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.899026    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmjl\" (UniqueName: \"kubernetes.io/projected/e2e89261-d1af-4dd1-af81-203e880592ee-kube-api-access-nqmjl\") pod \"keda-admission-5fbd5c4644-m7mwg\" (UID: \"e2e89261-d1af-4dd1-af81-203e880592ee\") " pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg"
Jul 02 12:45:10.899283 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:45:10.899115    2564 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 02 12:45:10.899283 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:45:10.899141    2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-m7mwg: secret "keda-admission-webhooks-certs" not found
Jul 02 12:45:10.899283 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:45:10.899206    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2e89261-d1af-4dd1-af81-203e880592ee-certificates podName:e2e89261-d1af-4dd1-af81-203e880592ee nodeName:}" failed. No retries permitted until 2026-07-02 12:45:11.399185068 +0000 UTC m=+359.382658733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e2e89261-d1af-4dd1-af81-203e880592ee-certificates") pod "keda-admission-5fbd5c4644-m7mwg" (UID: "e2e89261-d1af-4dd1-af81-203e880592ee") : secret "keda-admission-webhooks-certs" not found
Jul 02 12:45:10.901487 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.901468    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945-certificates\") pod \"keda-metrics-apiserver-559f75f947-2mps9\" (UID: \"5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:10.909344 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:10.909313    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmjl\" (UniqueName: \"kubernetes.io/projected/e2e89261-d1af-4dd1-af81-203e880592ee-kube-api-access-nqmjl\") pod \"keda-admission-5fbd5c4644-m7mwg\" (UID: \"e2e89261-d1af-4dd1-af81-203e880592ee\") " pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg"
Jul 02 12:45:11.026007 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:11.025898    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-dtrt9"]
Jul 02 12:45:11.028595 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:45:11.028564    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e28ae0_8984_461c_bf51_c0283231dc13.slice/crio-5959efc1b8bb1565b7b82f80527bee115aa4f6d395120b348835a0f96b0e7330 WatchSource:0}: Error finding container 5959efc1b8bb1565b7b82f80527bee115aa4f6d395120b348835a0f96b0e7330: Status 404 returned error can't find the container with id 5959efc1b8bb1565b7b82f80527bee115aa4f6d395120b348835a0f96b0e7330
Jul 02 12:45:11.124681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:11.124603    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:11.319116 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:11.319083    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"]
Jul 02 12:45:11.321557 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:45:11.321527    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5ac5c3_1d0d_4c3e_9e4d_8cbbcc227945.slice/crio-293091d2f2d2c146012fddac5756871dc11bf686b0fad680ad72dcfa61b1dd22 WatchSource:0}: Error finding container 293091d2f2d2c146012fddac5756871dc11bf686b0fad680ad72dcfa61b1dd22: Status 404 returned error can't find the container with id 293091d2f2d2c146012fddac5756871dc11bf686b0fad680ad72dcfa61b1dd22
Jul 02 12:45:11.403857 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:11.403776    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e2e89261-d1af-4dd1-af81-203e880592ee-certificates\") pod \"keda-admission-5fbd5c4644-m7mwg\" (UID: \"e2e89261-d1af-4dd1-af81-203e880592ee\") " pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg"
Jul 02 12:45:11.406195 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:11.406172    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e2e89261-d1af-4dd1-af81-203e880592ee-certificates\") pod \"keda-admission-5fbd5c4644-m7mwg\" (UID: \"e2e89261-d1af-4dd1-af81-203e880592ee\") " pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg"
Jul 02 12:45:11.644084 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:11.644046    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg"
Jul 02 12:45:11.808621 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:11.808439    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-m7mwg"]
Jul 02 12:45:11.811171 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:45:11.811146    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e89261_d1af_4dd1_af81_203e880592ee.slice/crio-2574df2aac6dd986a63d6d2e87e74a501f430394fcd77caf66e571ee7c2c26b4 WatchSource:0}: Error finding container 2574df2aac6dd986a63d6d2e87e74a501f430394fcd77caf66e571ee7c2c26b4: Status 404 returned error can't find the container with id 2574df2aac6dd986a63d6d2e87e74a501f430394fcd77caf66e571ee7c2c26b4
Jul 02 12:45:11.979501 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:11.979464    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg" event={"ID":"e2e89261-d1af-4dd1-af81-203e880592ee","Type":"ContainerStarted","Data":"2574df2aac6dd986a63d6d2e87e74a501f430394fcd77caf66e571ee7c2c26b4"}
Jul 02 12:45:11.980575 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:11.980546    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-dtrt9" event={"ID":"78e28ae0-8984-461c-bf51-c0283231dc13","Type":"ContainerStarted","Data":"5959efc1b8bb1565b7b82f80527bee115aa4f6d395120b348835a0f96b0e7330"}
Jul 02 12:45:11.981577 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:11.981558    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9" event={"ID":"5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945","Type":"ContainerStarted","Data":"293091d2f2d2c146012fddac5756871dc11bf686b0fad680ad72dcfa61b1dd22"}
Jul 02 12:45:13.995883 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:13.995840    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg" event={"ID":"e2e89261-d1af-4dd1-af81-203e880592ee","Type":"ContainerStarted","Data":"0393df7e88051139360d866317a06a667fe995b388808baae7f51c2266953698"}
Jul 02 12:45:13.996375 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:13.995994    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg"
Jul 02 12:45:14.028819 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:14.028762    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg" podStartSLOduration=1.978673906 podStartE2EDuration="4.028742803s" totalImagesPullingTime="2.050068897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:45:14.028436694 +0000 UTC m=+362.011910378" watchObservedRunningTime="2026-07-02 12:45:14.028742803 +0000 UTC m=+362.012216483"
Jul 02 12:45:17.010243 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:17.010204    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-dtrt9" event={"ID":"78e28ae0-8984-461c-bf51-c0283231dc13","Type":"ContainerStarted","Data":"bb46bc9d62ca44a365450ed2ff17f0b5e83505c1d4f2c428b69308ca0d607113"}
Jul 02 12:45:17.010685 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:17.010344    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:45:17.011772 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:17.011740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9" event={"ID":"5c5ac5c3-1d0d-4c3e-9e4d-8cbbcc227945","Type":"ContainerStarted","Data":"5542541010ed9d5ddd4b495905e7ef522d6273bd177f141dcf3fbf55ab807d30"}
Jul 02 12:45:17.011897 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:17.011856    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:17.036916 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:17.036872    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-dtrt9" podStartSLOduration=2.997769263 podStartE2EDuration="8.036859424s" totalImagesPullingTime="5.039090161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:45:17.03009462 +0000 UTC m=+365.013568303" watchObservedRunningTime="2026-07-02 12:45:17.036859424 +0000 UTC m=+365.020333106"
Jul 02 12:45:17.054724 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:17.054684    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9" podStartSLOduration=2.312893081 podStartE2EDuration="7.054669394s" totalImagesPullingTime="4.741776313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:45:17.053797742 +0000 UTC m=+365.037271424" watchObservedRunningTime="2026-07-02 12:45:17.054669394 +0000 UTC m=+365.038143096"
Jul 02 12:45:28.019598 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:28.019569    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-2mps9"
Jul 02 12:45:30.976696 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:30.976669    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-z6v7k"
Jul 02 12:45:35.002722 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:35.002692    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-m7mwg"
Jul 02 12:45:38.017586 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:45:38.017559    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-dtrt9"
Jul 02 12:46:45.228783 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.228705    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-jrddf"]
Jul 02 12:46:45.232023 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.231993    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:46:45.232705 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.232685    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-7hskt"]
Jul 02 12:46:45.234526 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.234502    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 12:46:45.234640 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.234552    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 02 12:46:45.235797 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.235778    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-2cdkk\""
Jul 02 12:46:45.235922 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.235841    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 12:46:45.235983 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.235970    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt"
Jul 02 12:46:45.238235 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.238218    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 02 12:46:45.238418 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.238399    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-4ftjq\""
Jul 02 12:46:45.246082 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.246060    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-jrddf"]
Jul 02 12:46:45.251396 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.251365    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-7hskt"]
Jul 02 12:46:45.284636 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.284607    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rrmv\" (UniqueName: \"kubernetes.io/projected/6b8e1c50-765f-43fa-b37f-e75bbbd8d879-kube-api-access-2rrmv\") pod \"llmisvc-controller-manager-67d95fdc48-7hskt\" (UID: \"6b8e1c50-765f-43fa-b37f-e75bbbd8d879\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt"
Jul 02 12:46:45.284821 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.284678    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b8e1c50-765f-43fa-b37f-e75bbbd8d879-cert\") pod \"llmisvc-controller-manager-67d95fdc48-7hskt\" (UID: \"6b8e1c50-765f-43fa-b37f-e75bbbd8d879\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt"
Jul 02 12:46:45.284821 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.284708    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxzfb\" (UniqueName: \"kubernetes.io/projected/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-kube-api-access-fxzfb\") pod \"kserve-controller-manager-69b5d94596-jrddf\" (UID: \"1dbfde79-0a62-4f44-8de8-cb4fedf120a9\") " pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:46:45.284821 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.284741    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-cert\") pod \"kserve-controller-manager-69b5d94596-jrddf\" (UID: \"1dbfde79-0a62-4f44-8de8-cb4fedf120a9\") " pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:46:45.385486 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.385449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b8e1c50-765f-43fa-b37f-e75bbbd8d879-cert\") pod \"llmisvc-controller-manager-67d95fdc48-7hskt\" (UID: \"6b8e1c50-765f-43fa-b37f-e75bbbd8d879\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt"
Jul 02 12:46:45.385486 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.385490    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxzfb\" (UniqueName: \"kubernetes.io/projected/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-kube-api-access-fxzfb\") pod \"kserve-controller-manager-69b5d94596-jrddf\" (UID: \"1dbfde79-0a62-4f44-8de8-cb4fedf120a9\") " pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:46:45.385739 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.385530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-cert\") pod \"kserve-controller-manager-69b5d94596-jrddf\" (UID: \"1dbfde79-0a62-4f44-8de8-cb4fedf120a9\") " pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:46:45.385739 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.385592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rrmv\" (UniqueName: \"kubernetes.io/projected/6b8e1c50-765f-43fa-b37f-e75bbbd8d879-kube-api-access-2rrmv\") pod \"llmisvc-controller-manager-67d95fdc48-7hskt\" (UID: \"6b8e1c50-765f-43fa-b37f-e75bbbd8d879\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt"
Jul 02 12:46:45.385739 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:46:45.385616    2564 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 02 12:46:45.385739 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:46:45.385676    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8e1c50-765f-43fa-b37f-e75bbbd8d879-cert podName:6b8e1c50-765f-43fa-b37f-e75bbbd8d879 nodeName:}" failed. No retries permitted until 2026-07-02 12:46:45.885654855 +0000 UTC m=+453.869128532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6b8e1c50-765f-43fa-b37f-e75bbbd8d879-cert") pod "llmisvc-controller-manager-67d95fdc48-7hskt" (UID: "6b8e1c50-765f-43fa-b37f-e75bbbd8d879") : secret "llmisvc-webhook-server-cert" not found
Jul 02 12:46:45.385739 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:46:45.385734    2564 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 02 12:46:45.385975 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:46:45.385808    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-cert podName:1dbfde79-0a62-4f44-8de8-cb4fedf120a9 nodeName:}" failed. No retries permitted until 2026-07-02 12:46:45.885790899 +0000 UTC m=+453.869264562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-cert") pod "kserve-controller-manager-69b5d94596-jrddf" (UID: "1dbfde79-0a62-4f44-8de8-cb4fedf120a9") : secret "kserve-webhook-server-cert" not found
Jul 02 12:46:45.394527 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.394505    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxzfb\" (UniqueName: \"kubernetes.io/projected/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-kube-api-access-fxzfb\") pod \"kserve-controller-manager-69b5d94596-jrddf\" (UID: \"1dbfde79-0a62-4f44-8de8-cb4fedf120a9\") " pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:46:45.395017 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.395000    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rrmv\" (UniqueName: \"kubernetes.io/projected/6b8e1c50-765f-43fa-b37f-e75bbbd8d879-kube-api-access-2rrmv\") pod \"llmisvc-controller-manager-67d95fdc48-7hskt\" (UID: \"6b8e1c50-765f-43fa-b37f-e75bbbd8d879\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt"
Jul 02 12:46:45.889381 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.889349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b8e1c50-765f-43fa-b37f-e75bbbd8d879-cert\") pod \"llmisvc-controller-manager-67d95fdc48-7hskt\" (UID: \"6b8e1c50-765f-43fa-b37f-e75bbbd8d879\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt"
Jul 02 12:46:45.889619 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.889398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-cert\") pod \"kserve-controller-manager-69b5d94596-jrddf\" (UID: \"1dbfde79-0a62-4f44-8de8-cb4fedf120a9\") " pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:46:45.891828 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.891798    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-cert\") pod \"kserve-controller-manager-69b5d94596-jrddf\" (UID: \"1dbfde79-0a62-4f44-8de8-cb4fedf120a9\") " pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:46:45.891927 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:45.891874    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b8e1c50-765f-43fa-b37f-e75bbbd8d879-cert\") pod \"llmisvc-controller-manager-67d95fdc48-7hskt\" (UID: \"6b8e1c50-765f-43fa-b37f-e75bbbd8d879\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt"
Jul 02 12:46:46.143505 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:46.143408    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:46:46.150631 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:46.150614    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt"
Jul 02 12:46:46.284247 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:46.284204    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-jrddf"]
Jul 02 12:46:46.287051 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:46:46.287024    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbfde79_0a62_4f44_8de8_cb4fedf120a9.slice/crio-f5123ee28fa7b7ea2906a505eee4e1cca0abf4c1ea1fe4c74ee52bd1f81ae351 WatchSource:0}: Error finding container f5123ee28fa7b7ea2906a505eee4e1cca0abf4c1ea1fe4c74ee52bd1f81ae351: Status 404 returned error can't find the container with id f5123ee28fa7b7ea2906a505eee4e1cca0abf4c1ea1fe4c74ee52bd1f81ae351
Jul 02 12:46:46.301467 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:46.301446    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-7hskt"]
Jul 02 12:46:46.303405 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:46:46.303382    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6b8e1c50_765f_43fa_b37f_e75bbbd8d879.slice/crio-ebc5f08f0607aee6222cb07df8898c590aecf19dfdf4106381183e0e60ca5347 WatchSource:0}: Error finding container ebc5f08f0607aee6222cb07df8898c590aecf19dfdf4106381183e0e60ca5347: Status 404 returned error can't find the container with id ebc5f08f0607aee6222cb07df8898c590aecf19dfdf4106381183e0e60ca5347
Jul 02 12:46:46.382526 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:46.382485    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt" event={"ID":"6b8e1c50-765f-43fa-b37f-e75bbbd8d879","Type":"ContainerStarted","Data":"ebc5f08f0607aee6222cb07df8898c590aecf19dfdf4106381183e0e60ca5347"}
Jul 02 12:46:46.383483 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:46.383461    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69b5d94596-jrddf" event={"ID":"1dbfde79-0a62-4f44-8de8-cb4fedf120a9","Type":"ContainerStarted","Data":"f5123ee28fa7b7ea2906a505eee4e1cca0abf4c1ea1fe4c74ee52bd1f81ae351"}
Jul 02 12:46:49.403241 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:49.403206    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt" event={"ID":"6b8e1c50-765f-43fa-b37f-e75bbbd8d879","Type":"ContainerStarted","Data":"6add3b01057e1ac9422df76808accdc5a8dc921945f4c03256248873ff934bc1"}
Jul 02 12:46:49.403695 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:49.403315    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt"
Jul 02 12:46:49.423156 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:49.423098    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt" podStartSLOduration=2.252607265 podStartE2EDuration="4.423077483s" totalImagesPullingTime="2.170470218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:46:49.421249993 +0000 UTC m=+457.404723676" watchObservedRunningTime="2026-07-02 12:46:49.423077483 +0000 UTC m=+457.406551167"
Jul 02 12:46:50.408566 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:50.408519    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69b5d94596-jrddf" event={"ID":"1dbfde79-0a62-4f44-8de8-cb4fedf120a9","Type":"ContainerStarted","Data":"e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd"}
Jul 02 12:46:50.409025 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:50.408735    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:46:50.428314 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:46:50.428259    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-69b5d94596-jrddf" podStartSLOduration=1.871264478 podStartE2EDuration="5.428243914s" totalImagesPullingTime="3.556979436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:46:50.426732388 +0000 UTC m=+458.410206070" watchObservedRunningTime="2026-07-02 12:46:50.428243914 +0000 UTC m=+458.411717596"
Jul 02 12:47:20.411461 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:20.411431    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-7hskt"
Jul 02 12:47:21.418544 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:21.418515    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:47:21.841075 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:21.841040    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-jrddf"]
Jul 02 12:47:21.841352 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:21.841291    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-69b5d94596-jrddf" podUID="1dbfde79-0a62-4f44-8de8-cb4fedf120a9" containerName="manager" containerID="cri-o://e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd" gracePeriod=10
Jul 02 12:47:22.081834 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.081809    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:47:22.205101 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.205064    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxzfb\" (UniqueName: \"kubernetes.io/projected/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-kube-api-access-fxzfb\") pod \"1dbfde79-0a62-4f44-8de8-cb4fedf120a9\" (UID: \"1dbfde79-0a62-4f44-8de8-cb4fedf120a9\") "
Jul 02 12:47:22.205269 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.205124    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-cert\") pod \"1dbfde79-0a62-4f44-8de8-cb4fedf120a9\" (UID: \"1dbfde79-0a62-4f44-8de8-cb4fedf120a9\") "
Jul 02 12:47:22.207137 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.207106    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-cert" (OuterVolumeSpecName: "cert") pod "1dbfde79-0a62-4f44-8de8-cb4fedf120a9" (UID: "1dbfde79-0a62-4f44-8de8-cb4fedf120a9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:47:22.207239 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.207157    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-kube-api-access-fxzfb" (OuterVolumeSpecName: "kube-api-access-fxzfb") pod "1dbfde79-0a62-4f44-8de8-cb4fedf120a9" (UID: "1dbfde79-0a62-4f44-8de8-cb4fedf120a9"). InnerVolumeSpecName "kube-api-access-fxzfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:47:22.306592 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.306562    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxzfb\" (UniqueName: \"kubernetes.io/projected/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-kube-api-access-fxzfb\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:47:22.306592 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.306587    2564 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbfde79-0a62-4f44-8de8-cb4fedf120a9-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:47:22.544182 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.544115    2564 generic.go:356] "Generic (PLEG): container finished" podID="1dbfde79-0a62-4f44-8de8-cb4fedf120a9" containerID="e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd" exitCode=0
Jul 02 12:47:22.544182 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.544158    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69b5d94596-jrddf"
Jul 02 12:47:22.544607 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.544181    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69b5d94596-jrddf" event={"ID":"1dbfde79-0a62-4f44-8de8-cb4fedf120a9","Type":"ContainerDied","Data":"e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd"}
Jul 02 12:47:22.544607 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.544241    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69b5d94596-jrddf" event={"ID":"1dbfde79-0a62-4f44-8de8-cb4fedf120a9","Type":"ContainerDied","Data":"f5123ee28fa7b7ea2906a505eee4e1cca0abf4c1ea1fe4c74ee52bd1f81ae351"}
Jul 02 12:47:22.544607 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.544262    2564 scope.go:117] "RemoveContainer" containerID="e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd"
Jul 02 12:47:22.555211 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.555195    2564 scope.go:117] "RemoveContainer" containerID="e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd"
Jul 02 12:47:22.555478 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:47:22.555457    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd\": container with ID starting with e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd not found: ID does not exist" containerID="e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd"
Jul 02 12:47:22.555579 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.555485    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd"} err="failed to get container status \"e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd\": rpc error: code = NotFound desc = could not find container \"e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd\": container with ID starting with e6ffb8ae856afd6890a9ab559ce251d37224daa0ba7aadfae29453c8f9eec7bd not found: ID does not exist"
Jul 02 12:47:22.560867 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.560847    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-jrddf"]
Jul 02 12:47:22.564859 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:22.564839    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-69b5d94596-jrddf"]
Jul 02 12:47:23.626476 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.626442    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d8d54b864-gt52k"]
Jul 02 12:47:23.626913 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.626740    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbfde79-0a62-4f44-8de8-cb4fedf120a9" containerName="manager"
Jul 02 12:47:23.626913 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.626751    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbfde79-0a62-4f44-8de8-cb4fedf120a9" containerName="manager"
Jul 02 12:47:23.626913 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.626812    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dbfde79-0a62-4f44-8de8-cb4fedf120a9" containerName="manager"
Jul 02 12:47:23.674345 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.674314    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d8d54b864-gt52k"]
Jul 02 12:47:23.674491 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.674405    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.821051 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.821018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mgbv\" (UniqueName: \"kubernetes.io/projected/5fdf7d99-262b-4054-af0c-12524be97fc8-kube-api-access-5mgbv\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.821051 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.821051    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fdf7d99-262b-4054-af0c-12524be97fc8-console-config\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.821339 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.821078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fdf7d99-262b-4054-af0c-12524be97fc8-console-oauth-config\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.821339 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.821105    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fdf7d99-262b-4054-af0c-12524be97fc8-trusted-ca-bundle\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.821339 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.821179    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fdf7d99-262b-4054-af0c-12524be97fc8-oauth-serving-cert\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.821339 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.821240    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdf7d99-262b-4054-af0c-12524be97fc8-service-ca\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.821339 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.821287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fdf7d99-262b-4054-af0c-12524be97fc8-console-serving-cert\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.922406 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.922321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdf7d99-262b-4054-af0c-12524be97fc8-service-ca\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.922406 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.922390    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fdf7d99-262b-4054-af0c-12524be97fc8-console-serving-cert\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.922605 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.922446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mgbv\" (UniqueName: \"kubernetes.io/projected/5fdf7d99-262b-4054-af0c-12524be97fc8-kube-api-access-5mgbv\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.922605 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.922473    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fdf7d99-262b-4054-af0c-12524be97fc8-console-config\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.922605 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.922513    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fdf7d99-262b-4054-af0c-12524be97fc8-console-oauth-config\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.922605 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.922536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fdf7d99-262b-4054-af0c-12524be97fc8-trusted-ca-bundle\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.922605 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.922565    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fdf7d99-262b-4054-af0c-12524be97fc8-oauth-serving-cert\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.923148 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.923122    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fdf7d99-262b-4054-af0c-12524be97fc8-console-config\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.923262 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.923127    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdf7d99-262b-4054-af0c-12524be97fc8-service-ca\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.923343 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.923322    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fdf7d99-262b-4054-af0c-12524be97fc8-oauth-serving-cert\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.923653 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.923630    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fdf7d99-262b-4054-af0c-12524be97fc8-trusted-ca-bundle\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.924937 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.924914    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fdf7d99-262b-4054-af0c-12524be97fc8-console-oauth-config\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.925067 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.925047    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fdf7d99-262b-4054-af0c-12524be97fc8-console-serving-cert\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.932350 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.932328    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mgbv\" (UniqueName: \"kubernetes.io/projected/5fdf7d99-262b-4054-af0c-12524be97fc8-kube-api-access-5mgbv\") pod \"console-5d8d54b864-gt52k\" (UID: \"5fdf7d99-262b-4054-af0c-12524be97fc8\") " pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:23.983287 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:23.983248    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:24.144020 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:24.143999    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d8d54b864-gt52k"]
Jul 02 12:47:24.146648 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:47:24.146618    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdf7d99_262b_4054_af0c_12524be97fc8.slice/crio-a8b9636851b2157e0e4c243a037fd22b419d643cd2e9b4374df23f5e333a5df5 WatchSource:0}: Error finding container a8b9636851b2157e0e4c243a037fd22b419d643cd2e9b4374df23f5e333a5df5: Status 404 returned error can't find the container with id a8b9636851b2157e0e4c243a037fd22b419d643cd2e9b4374df23f5e333a5df5
Jul 02 12:47:24.422662 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:24.422632    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dbfde79-0a62-4f44-8de8-cb4fedf120a9" path="/var/lib/kubelet/pods/1dbfde79-0a62-4f44-8de8-cb4fedf120a9/volumes"
Jul 02 12:47:24.554965 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:24.554931    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8d54b864-gt52k" event={"ID":"5fdf7d99-262b-4054-af0c-12524be97fc8","Type":"ContainerStarted","Data":"b3590181d8d96b86f831d0541f4da9295ffe17c97adf8d613faee6fffb3fa64c"}
Jul 02 12:47:24.555177 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:24.554991    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8d54b864-gt52k" event={"ID":"5fdf7d99-262b-4054-af0c-12524be97fc8","Type":"ContainerStarted","Data":"a8b9636851b2157e0e4c243a037fd22b419d643cd2e9b4374df23f5e333a5df5"}
Jul 02 12:47:24.574178 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:24.574129    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d8d54b864-gt52k" podStartSLOduration=1.574113981 podStartE2EDuration="1.574113981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:47:24.572637313 +0000 UTC m=+492.556110996" watchObservedRunningTime="2026-07-02 12:47:24.574113981 +0000 UTC m=+492.557587664"
Jul 02 12:47:33.984077 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:33.984041    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:33.984667 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:33.984110    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:33.989149 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:33.989126    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:34.600120 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:34.600094    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d8d54b864-gt52k"
Jul 02 12:47:34.649853 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:34.649820    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b7cbc7b7-wdtrg"]
Jul 02 12:47:56.956848 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:56.956809    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-9n5jl"]
Jul 02 12:47:56.960635 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:56.960613    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-9n5jl"
Jul 02 12:47:56.963182 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:56.963154    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 02 12:47:56.963455 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:56.963441    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-zpgsp\""
Jul 02 12:47:56.971266 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:56.971238    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-9n5jl"]
Jul 02 12:47:57.096320 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:57.096246    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq89g\" (UniqueName: \"kubernetes.io/projected/d86752f9-b757-4fd9-a401-b39000847b11-kube-api-access-bq89g\") pod \"model-serving-api-fd65d7d6-9n5jl\" (UID: \"d86752f9-b757-4fd9-a401-b39000847b11\") " pod="kserve/model-serving-api-fd65d7d6-9n5jl"
Jul 02 12:47:57.096502 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:57.096344    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86752f9-b757-4fd9-a401-b39000847b11-tls-certs\") pod \"model-serving-api-fd65d7d6-9n5jl\" (UID: \"d86752f9-b757-4fd9-a401-b39000847b11\") " pod="kserve/model-serving-api-fd65d7d6-9n5jl"
Jul 02 12:47:57.197379 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:57.197334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq89g\" (UniqueName: \"kubernetes.io/projected/d86752f9-b757-4fd9-a401-b39000847b11-kube-api-access-bq89g\") pod \"model-serving-api-fd65d7d6-9n5jl\" (UID: \"d86752f9-b757-4fd9-a401-b39000847b11\") " pod="kserve/model-serving-api-fd65d7d6-9n5jl"
Jul 02 12:47:57.197564 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:57.197403    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86752f9-b757-4fd9-a401-b39000847b11-tls-certs\") pod \"model-serving-api-fd65d7d6-9n5jl\" (UID: \"d86752f9-b757-4fd9-a401-b39000847b11\") " pod="kserve/model-serving-api-fd65d7d6-9n5jl"
Jul 02 12:47:57.197638 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:47:57.197569    2564 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 02 12:47:57.197692 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:47:57.197662    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86752f9-b757-4fd9-a401-b39000847b11-tls-certs podName:d86752f9-b757-4fd9-a401-b39000847b11 nodeName:}" failed. No retries permitted until 2026-07-02 12:47:57.697637751 +0000 UTC m=+525.681111419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d86752f9-b757-4fd9-a401-b39000847b11-tls-certs") pod "model-serving-api-fd65d7d6-9n5jl" (UID: "d86752f9-b757-4fd9-a401-b39000847b11") : secret "model-serving-api-tls" not found
Jul 02 12:47:57.213822 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:57.213755    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq89g\" (UniqueName: \"kubernetes.io/projected/d86752f9-b757-4fd9-a401-b39000847b11-kube-api-access-bq89g\") pod \"model-serving-api-fd65d7d6-9n5jl\" (UID: \"d86752f9-b757-4fd9-a401-b39000847b11\") " pod="kserve/model-serving-api-fd65d7d6-9n5jl"
Jul 02 12:47:57.702162 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:57.702126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86752f9-b757-4fd9-a401-b39000847b11-tls-certs\") pod \"model-serving-api-fd65d7d6-9n5jl\" (UID: \"d86752f9-b757-4fd9-a401-b39000847b11\") " pod="kserve/model-serving-api-fd65d7d6-9n5jl"
Jul 02 12:47:57.704463 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:57.704443    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86752f9-b757-4fd9-a401-b39000847b11-tls-certs\") pod \"model-serving-api-fd65d7d6-9n5jl\" (UID: \"d86752f9-b757-4fd9-a401-b39000847b11\") " pod="kserve/model-serving-api-fd65d7d6-9n5jl"
Jul 02 12:47:57.873176 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:57.873138    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-9n5jl"
Jul 02 12:47:58.001211 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:58.001140    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-9n5jl"]
Jul 02 12:47:58.003939 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:47:58.003886    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86752f9_b757_4fd9_a401_b39000847b11.slice/crio-74104eefab5920f7f122eeefd1a1e399e29876a63078d6784e22d6832da87078 WatchSource:0}: Error finding container 74104eefab5920f7f122eeefd1a1e399e29876a63078d6784e22d6832da87078: Status 404 returned error can't find the container with id 74104eefab5920f7f122eeefd1a1e399e29876a63078d6784e22d6832da87078
Jul 02 12:47:58.693787 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:58.693752    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-9n5jl" event={"ID":"d86752f9-b757-4fd9-a401-b39000847b11","Type":"ContainerStarted","Data":"74104eefab5920f7f122eeefd1a1e399e29876a63078d6784e22d6832da87078"}
Jul 02 12:47:59.678225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:59.678187    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-8b7cbc7b7-wdtrg" podUID="0921b634-18dd-43b4-90b2-234a7d7a5a9b" containerName="console" containerID="cri-o://89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc" gracePeriod=15
Jul 02 12:47:59.938824 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:47:59.938803    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:48:00.023227 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023196    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-config\") pod \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") "
Jul 02 12:48:00.023432 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023245    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-oauth-serving-cert\") pod \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") "
Jul 02 12:48:00.023432 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023271    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-serving-cert\") pod \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") "
Jul 02 12:48:00.023432 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023316    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-trusted-ca-bundle\") pod \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") "
Jul 02 12:48:00.023432 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023336    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-service-ca\") pod \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") "
Jul 02 12:48:00.023432 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023373    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4b2r\" (UniqueName: \"kubernetes.io/projected/0921b634-18dd-43b4-90b2-234a7d7a5a9b-kube-api-access-z4b2r\") pod \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") "
Jul 02 12:48:00.023432 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023418    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-oauth-config\") pod \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\" (UID: \"0921b634-18dd-43b4-90b2-234a7d7a5a9b\") "
Jul 02 12:48:00.023730 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023649    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-config" (OuterVolumeSpecName: "console-config") pod "0921b634-18dd-43b4-90b2-234a7d7a5a9b" (UID: "0921b634-18dd-43b4-90b2-234a7d7a5a9b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:48:00.023730 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023681    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-service-ca" (OuterVolumeSpecName: "service-ca") pod "0921b634-18dd-43b4-90b2-234a7d7a5a9b" (UID: "0921b634-18dd-43b4-90b2-234a7d7a5a9b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:48:00.023730 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023700    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0921b634-18dd-43b4-90b2-234a7d7a5a9b" (UID: "0921b634-18dd-43b4-90b2-234a7d7a5a9b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:48:00.023870 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023754    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0921b634-18dd-43b4-90b2-234a7d7a5a9b" (UID: "0921b634-18dd-43b4-90b2-234a7d7a5a9b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:48:00.023870 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023760    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:48:00.023870 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.023806    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-service-ca\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:48:00.025487 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.025466    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0921b634-18dd-43b4-90b2-234a7d7a5a9b" (UID: "0921b634-18dd-43b4-90b2-234a7d7a5a9b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:48:00.025561 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.025508    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0921b634-18dd-43b4-90b2-234a7d7a5a9b-kube-api-access-z4b2r" (OuterVolumeSpecName: "kube-api-access-z4b2r") pod "0921b634-18dd-43b4-90b2-234a7d7a5a9b" (UID: "0921b634-18dd-43b4-90b2-234a7d7a5a9b"). InnerVolumeSpecName "kube-api-access-z4b2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:48:00.025695 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.025667    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0921b634-18dd-43b4-90b2-234a7d7a5a9b" (UID: "0921b634-18dd-43b4-90b2-234a7d7a5a9b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:48:00.124233 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.124179    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-oauth-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:48:00.124233 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.124225    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:48:00.124233 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.124235    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0921b634-18dd-43b4-90b2-234a7d7a5a9b-trusted-ca-bundle\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:48:00.124233 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.124246    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4b2r\" (UniqueName: \"kubernetes.io/projected/0921b634-18dd-43b4-90b2-234a7d7a5a9b-kube-api-access-z4b2r\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:48:00.124233 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.124255    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0921b634-18dd-43b4-90b2-234a7d7a5a9b-console-oauth-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:48:00.704579 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.704545    2564 generic.go:356] "Generic (PLEG): container finished" podID="0921b634-18dd-43b4-90b2-234a7d7a5a9b" containerID="89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc" exitCode=0
Jul 02 12:48:00.704960 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.704606    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b7cbc7b7-wdtrg"
Jul 02 12:48:00.704960 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.704630    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b7cbc7b7-wdtrg" event={"ID":"0921b634-18dd-43b4-90b2-234a7d7a5a9b","Type":"ContainerDied","Data":"89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc"}
Jul 02 12:48:00.704960 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.704673    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b7cbc7b7-wdtrg" event={"ID":"0921b634-18dd-43b4-90b2-234a7d7a5a9b","Type":"ContainerDied","Data":"eb11c60eae48c5020bf1a90e2dcde9fb7642bc01e4c275ef658e72887caabb00"}
Jul 02 12:48:00.704960 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.704692    2564 scope.go:117] "RemoveContainer" containerID="89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc"
Jul 02 12:48:00.712842 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.712825    2564 scope.go:117] "RemoveContainer" containerID="89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc"
Jul 02 12:48:00.713134 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:48:00.713113    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc\": container with ID starting with 89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc not found: ID does not exist" containerID="89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc"
Jul 02 12:48:00.713235 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.713146    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc"} err="failed to get container status \"89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc\": rpc error: code = NotFound desc = could not find container \"89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc\": container with ID starting with 89789981cf408f4e96815a2a928c43c659b0976c85dd74089ab793ce0b3f02dc not found: ID does not exist"
Jul 02 12:48:00.721537 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.721508    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b7cbc7b7-wdtrg"]
Jul 02 12:48:00.723443 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:00.723422    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8b7cbc7b7-wdtrg"]
Jul 02 12:48:02.424045 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:02.423952    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0921b634-18dd-43b4-90b2-234a7d7a5a9b" path="/var/lib/kubelet/pods/0921b634-18dd-43b4-90b2-234a7d7a5a9b/volumes"
Jul 02 12:48:02.718031 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:02.717996    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-9n5jl" event={"ID":"d86752f9-b757-4fd9-a401-b39000847b11","Type":"ContainerStarted","Data":"511fea8615d4e71ccf4e1a943d42de47769e653b37c45f837662fb1e3c3c6924"}
Jul 02 12:48:02.718210 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:02.718082    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-9n5jl"
Jul 02 12:48:02.735093 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:02.735044    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-9n5jl" podStartSLOduration=2.764600061 podStartE2EDuration="6.735029877s" totalImagesPullingTime="3.970429816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:48:02.733982807 +0000 UTC m=+530.717456492" watchObservedRunningTime="2026-07-02 12:48:02.735029877 +0000 UTC m=+530.718503577"
Jul 02 12:48:13.726692 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:48:13.726662    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-9n5jl"
Jul 02 12:49:31.601090 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.601050    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"]
Jul 02 12:49:31.601559 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.601392    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0921b634-18dd-43b4-90b2-234a7d7a5a9b" containerName="console"
Jul 02 12:49:31.601559 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.601404    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0921b634-18dd-43b4-90b2-234a7d7a5a9b" containerName="console"
Jul 02 12:49:31.601559 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.601470    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0921b634-18dd-43b4-90b2-234a7d7a5a9b" containerName="console"
Jul 02 12:49:31.604340 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.604324    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"
Jul 02 12:49:31.606947 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.606927    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 02 12:49:31.607069 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.606934    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-5dprx\""
Jul 02 12:49:31.612709 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.612685    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"]
Jul 02 12:49:31.685573 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.685543    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrxwn\" (UniqueName: \"kubernetes.io/projected/b0cdc2c3-11b2-4501-8b7d-9842d8952d60-kube-api-access-hrxwn\") pod \"seaweedfs-tls-custom-6644bf5d6c-mbthj\" (UID: \"b0cdc2c3-11b2-4501-8b7d-9842d8952d60\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"
Jul 02 12:49:31.685707 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.685603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b0cdc2c3-11b2-4501-8b7d-9842d8952d60-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-mbthj\" (UID: \"b0cdc2c3-11b2-4501-8b7d-9842d8952d60\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"
Jul 02 12:49:31.787023 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.786983    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrxwn\" (UniqueName: \"kubernetes.io/projected/b0cdc2c3-11b2-4501-8b7d-9842d8952d60-kube-api-access-hrxwn\") pod \"seaweedfs-tls-custom-6644bf5d6c-mbthj\" (UID: \"b0cdc2c3-11b2-4501-8b7d-9842d8952d60\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"
Jul 02 12:49:31.787181 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.787047    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b0cdc2c3-11b2-4501-8b7d-9842d8952d60-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-mbthj\" (UID: \"b0cdc2c3-11b2-4501-8b7d-9842d8952d60\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"
Jul 02 12:49:31.787422 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.787403    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b0cdc2c3-11b2-4501-8b7d-9842d8952d60-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-mbthj\" (UID: \"b0cdc2c3-11b2-4501-8b7d-9842d8952d60\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"
Jul 02 12:49:31.796397 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.796374    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrxwn\" (UniqueName: \"kubernetes.io/projected/b0cdc2c3-11b2-4501-8b7d-9842d8952d60-kube-api-access-hrxwn\") pod \"seaweedfs-tls-custom-6644bf5d6c-mbthj\" (UID: \"b0cdc2c3-11b2-4501-8b7d-9842d8952d60\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"
Jul 02 12:49:31.914579 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:31.914522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"
Jul 02 12:49:32.043615 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:32.043590    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"]
Jul 02 12:49:32.046238 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:49:32.046211    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cdc2c3_11b2_4501_8b7d_9842d8952d60.slice/crio-e0b720c00543ecde0d941bea392e8feb3fec1249a377b0fbfdb62d0297e12713 WatchSource:0}: Error finding container e0b720c00543ecde0d941bea392e8feb3fec1249a377b0fbfdb62d0297e12713: Status 404 returned error can't find the container with id e0b720c00543ecde0d941bea392e8feb3fec1249a377b0fbfdb62d0297e12713
Jul 02 12:49:32.112655 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:32.112627    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj" event={"ID":"b0cdc2c3-11b2-4501-8b7d-9842d8952d60","Type":"ContainerStarted","Data":"e0b720c00543ecde0d941bea392e8feb3fec1249a377b0fbfdb62d0297e12713"}
Jul 02 12:49:36.131615 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:36.131575    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj" event={"ID":"b0cdc2c3-11b2-4501-8b7d-9842d8952d60","Type":"ContainerStarted","Data":"709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8"}
Jul 02 12:49:36.148810 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:36.148769    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj" podStartSLOduration=1.618475608 podStartE2EDuration="5.148753156s" totalImagesPullingTime="3.530277548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:49:36.147847711 +0000 UTC m=+624.131321404" watchObservedRunningTime="2026-07-02 12:49:36.148753156 +0000 UTC m=+624.132226837"
Jul 02 12:49:37.141461 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:37.141429    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"]
Jul 02 12:49:38.139337 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:38.139262    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj" podUID="b0cdc2c3-11b2-4501-8b7d-9842d8952d60" containerName="seaweedfs-tls-custom" containerID="cri-o://709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8" gracePeriod=30
Jul 02 12:49:39.381543 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:39.381522    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"
Jul 02 12:49:39.450879 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:39.450856    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrxwn\" (UniqueName: \"kubernetes.io/projected/b0cdc2c3-11b2-4501-8b7d-9842d8952d60-kube-api-access-hrxwn\") pod \"b0cdc2c3-11b2-4501-8b7d-9842d8952d60\" (UID: \"b0cdc2c3-11b2-4501-8b7d-9842d8952d60\") "
Jul 02 12:49:39.450997 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:39.450891    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b0cdc2c3-11b2-4501-8b7d-9842d8952d60-data\") pod \"b0cdc2c3-11b2-4501-8b7d-9842d8952d60\" (UID: \"b0cdc2c3-11b2-4501-8b7d-9842d8952d60\") "
Jul 02 12:49:39.452137 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:39.452115    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cdc2c3-11b2-4501-8b7d-9842d8952d60-data" (OuterVolumeSpecName: "data") pod "b0cdc2c3-11b2-4501-8b7d-9842d8952d60" (UID: "b0cdc2c3-11b2-4501-8b7d-9842d8952d60"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:49:39.452742 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:39.452720    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cdc2c3-11b2-4501-8b7d-9842d8952d60-kube-api-access-hrxwn" (OuterVolumeSpecName: "kube-api-access-hrxwn") pod "b0cdc2c3-11b2-4501-8b7d-9842d8952d60" (UID: "b0cdc2c3-11b2-4501-8b7d-9842d8952d60"). InnerVolumeSpecName "kube-api-access-hrxwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:49:39.551683 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:39.551654    2564 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b0cdc2c3-11b2-4501-8b7d-9842d8952d60-data\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:49:39.551683 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:39.551678    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrxwn\" (UniqueName: \"kubernetes.io/projected/b0cdc2c3-11b2-4501-8b7d-9842d8952d60-kube-api-access-hrxwn\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:49:40.148341 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.148290    2564 generic.go:356] "Generic (PLEG): container finished" podID="b0cdc2c3-11b2-4501-8b7d-9842d8952d60" containerID="709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8" exitCode=0
Jul 02 12:49:40.148519 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.148350    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"
Jul 02 12:49:40.148519 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.148390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj" event={"ID":"b0cdc2c3-11b2-4501-8b7d-9842d8952d60","Type":"ContainerDied","Data":"709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8"}
Jul 02 12:49:40.148519 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.148438    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj" event={"ID":"b0cdc2c3-11b2-4501-8b7d-9842d8952d60","Type":"ContainerDied","Data":"e0b720c00543ecde0d941bea392e8feb3fec1249a377b0fbfdb62d0297e12713"}
Jul 02 12:49:40.148519 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.148457    2564 scope.go:117] "RemoveContainer" containerID="709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8"
Jul 02 12:49:40.158419 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.158402    2564 scope.go:117] "RemoveContainer" containerID="709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8"
Jul 02 12:49:40.158694 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:49:40.158675    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8\": container with ID starting with 709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8 not found: ID does not exist" containerID="709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8"
Jul 02 12:49:40.158776 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.158706    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8"} err="failed to get container status \"709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8\": rpc error: code = NotFound desc = could not find container \"709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8\": container with ID starting with 709ce3b7b2a8cc5871f5e3cb0dda3f2bf7226f2b2ea9b5e3606e9e1703700fc8 not found: ID does not exist"
Jul 02 12:49:40.169373 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.169352    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"]
Jul 02 12:49:40.173257 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.173239    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mbthj"]
Jul 02 12:49:40.201685 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.201664    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-k8shf"]
Jul 02 12:49:40.201973 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.201961    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0cdc2c3-11b2-4501-8b7d-9842d8952d60" containerName="seaweedfs-tls-custom"
Jul 02 12:49:40.202014 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.201975    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0cdc2c3-11b2-4501-8b7d-9842d8952d60" containerName="seaweedfs-tls-custom"
Jul 02 12:49:40.202051 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.202040    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0cdc2c3-11b2-4501-8b7d-9842d8952d60" containerName="seaweedfs-tls-custom"
Jul 02 12:49:40.206406 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.206392    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf"
Jul 02 12:49:40.208861 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.208847    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 02 12:49:40.208928 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.208887    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-5dprx\""
Jul 02 12:49:40.208972 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.208887    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 02 12:49:40.211540 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.211522    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-k8shf"]
Jul 02 12:49:40.256737 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.256712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c6e83fc6-1212-41cb-8391-46204f20f25a-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-k8shf\" (UID: \"c6e83fc6-1212-41cb-8391-46204f20f25a\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf"
Jul 02 12:49:40.256825 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.256753    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c6e83fc6-1212-41cb-8391-46204f20f25a-data\") pod \"seaweedfs-tls-custom-8b44d767d-k8shf\" (UID: \"c6e83fc6-1212-41cb-8391-46204f20f25a\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf"
Jul 02 12:49:40.256877 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.256840    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjkcd\" (UniqueName: \"kubernetes.io/projected/c6e83fc6-1212-41cb-8391-46204f20f25a-kube-api-access-qjkcd\") pod \"seaweedfs-tls-custom-8b44d767d-k8shf\" (UID: \"c6e83fc6-1212-41cb-8391-46204f20f25a\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf"
Jul 02 12:49:40.357379 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.357357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c6e83fc6-1212-41cb-8391-46204f20f25a-data\") pod \"seaweedfs-tls-custom-8b44d767d-k8shf\" (UID: \"c6e83fc6-1212-41cb-8391-46204f20f25a\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf"
Jul 02 12:49:40.357505 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.357409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjkcd\" (UniqueName: \"kubernetes.io/projected/c6e83fc6-1212-41cb-8391-46204f20f25a-kube-api-access-qjkcd\") pod \"seaweedfs-tls-custom-8b44d767d-k8shf\" (UID: \"c6e83fc6-1212-41cb-8391-46204f20f25a\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf"
Jul 02 12:49:40.357505 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.357450    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c6e83fc6-1212-41cb-8391-46204f20f25a-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-k8shf\" (UID: \"c6e83fc6-1212-41cb-8391-46204f20f25a\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf"
Jul 02 12:49:40.357749 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.357732    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c6e83fc6-1212-41cb-8391-46204f20f25a-data\") pod \"seaweedfs-tls-custom-8b44d767d-k8shf\" (UID: \"c6e83fc6-1212-41cb-8391-46204f20f25a\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf"
Jul 02 12:49:40.359873 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.359852    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c6e83fc6-1212-41cb-8391-46204f20f25a-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-k8shf\" (UID: \"c6e83fc6-1212-41cb-8391-46204f20f25a\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf"
Jul 02 12:49:40.365868 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.365846    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjkcd\" (UniqueName: \"kubernetes.io/projected/c6e83fc6-1212-41cb-8391-46204f20f25a-kube-api-access-qjkcd\") pod \"seaweedfs-tls-custom-8b44d767d-k8shf\" (UID: \"c6e83fc6-1212-41cb-8391-46204f20f25a\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf"
Jul 02 12:49:40.422436 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.422375    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0cdc2c3-11b2-4501-8b7d-9842d8952d60" path="/var/lib/kubelet/pods/b0cdc2c3-11b2-4501-8b7d-9842d8952d60/volumes"
Jul 02 12:49:40.516631 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.516610    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf"
Jul 02 12:49:40.645162 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:40.645138    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-k8shf"]
Jul 02 12:49:40.647817 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:49:40.647790    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e83fc6_1212_41cb_8391_46204f20f25a.slice/crio-260ee66adf3e44a3263dcff9644254fdbd19d924f255f7f5856761e579d41110 WatchSource:0}: Error finding container 260ee66adf3e44a3263dcff9644254fdbd19d924f255f7f5856761e579d41110: Status 404 returned error can't find the container with id 260ee66adf3e44a3263dcff9644254fdbd19d924f255f7f5856761e579d41110
Jul 02 12:49:41.155860 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:41.155813    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf" event={"ID":"c6e83fc6-1212-41cb-8391-46204f20f25a","Type":"ContainerStarted","Data":"33ff0307598b96f51e4390a8d0fd0c62616d250af6a78c1ff11a2512f39bb9ab"}
Jul 02 12:49:41.156039 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:41.155883    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf" event={"ID":"c6e83fc6-1212-41cb-8391-46204f20f25a","Type":"ContainerStarted","Data":"260ee66adf3e44a3263dcff9644254fdbd19d924f255f7f5856761e579d41110"}
Jul 02 12:49:41.175315 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:41.175202    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-k8shf" podStartSLOduration=0.911383137 podStartE2EDuration="1.175189105s" totalImagesPullingTime="263.805968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:49:41.173746095 +0000 UTC m=+629.157219779" watchObservedRunningTime="2026-07-02 12:49:41.175189105 +0000 UTC m=+629.158662787"
Jul 02 12:49:53.598225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.598194    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"]
Jul 02 12:49:53.602067 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.602049    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"
Jul 02 12:49:53.604683 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.604656    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 02 12:49:53.604683 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.604671    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 02 12:49:53.612817 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.612789    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"]
Jul 02 12:49:53.674246 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.674205    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/94f32a34-da9d-491d-aeaf-062ce775664d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-gd68j\" (UID: \"94f32a34-da9d-491d-aeaf-062ce775664d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"
Jul 02 12:49:53.674246 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.674250    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/94f32a34-da9d-491d-aeaf-062ce775664d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-gd68j\" (UID: \"94f32a34-da9d-491d-aeaf-062ce775664d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"
Jul 02 12:49:53.674478 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.674389    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sfjm\" (UniqueName: \"kubernetes.io/projected/94f32a34-da9d-491d-aeaf-062ce775664d-kube-api-access-7sfjm\") pod \"seaweedfs-tls-serving-67b7477c6d-gd68j\" (UID: \"94f32a34-da9d-491d-aeaf-062ce775664d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"
Jul 02 12:49:53.775580 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.775542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/94f32a34-da9d-491d-aeaf-062ce775664d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-gd68j\" (UID: \"94f32a34-da9d-491d-aeaf-062ce775664d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"
Jul 02 12:49:53.775752 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.775639    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sfjm\" (UniqueName: \"kubernetes.io/projected/94f32a34-da9d-491d-aeaf-062ce775664d-kube-api-access-7sfjm\") pod \"seaweedfs-tls-serving-67b7477c6d-gd68j\" (UID: \"94f32a34-da9d-491d-aeaf-062ce775664d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"
Jul 02 12:49:53.775752 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.775699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/94f32a34-da9d-491d-aeaf-062ce775664d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-gd68j\" (UID: \"94f32a34-da9d-491d-aeaf-062ce775664d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"
Jul 02 12:49:53.775895 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:49:53.775802    2564 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 02 12:49:53.775895 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:49:53.775821    2564 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-gd68j: secret "seaweedfs-tls-serving" not found
Jul 02 12:49:53.775895 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:49:53.775890    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94f32a34-da9d-491d-aeaf-062ce775664d-seaweedfs-tls-serving podName:94f32a34-da9d-491d-aeaf-062ce775664d nodeName:}" failed. No retries permitted until 2026-07-02 12:49:54.275868911 +0000 UTC m=+642.259342572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/94f32a34-da9d-491d-aeaf-062ce775664d-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-gd68j" (UID: "94f32a34-da9d-491d-aeaf-062ce775664d") : secret "seaweedfs-tls-serving" not found
Jul 02 12:49:53.776026 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.775971    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/94f32a34-da9d-491d-aeaf-062ce775664d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-gd68j\" (UID: \"94f32a34-da9d-491d-aeaf-062ce775664d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"
Jul 02 12:49:53.784755 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:53.784725    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sfjm\" (UniqueName: \"kubernetes.io/projected/94f32a34-da9d-491d-aeaf-062ce775664d-kube-api-access-7sfjm\") pod \"seaweedfs-tls-serving-67b7477c6d-gd68j\" (UID: \"94f32a34-da9d-491d-aeaf-062ce775664d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"
Jul 02 12:49:54.279897 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:54.279858    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/94f32a34-da9d-491d-aeaf-062ce775664d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-gd68j\" (UID: \"94f32a34-da9d-491d-aeaf-062ce775664d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"
Jul 02 12:49:54.282278 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:54.282241    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/94f32a34-da9d-491d-aeaf-062ce775664d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-gd68j\" (UID: \"94f32a34-da9d-491d-aeaf-062ce775664d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"
Jul 02 12:49:54.511948 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:54.511907    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"
Jul 02 12:49:54.639394 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:54.639365    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-gd68j"]
Jul 02 12:49:54.642424 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:49:54.642398    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f32a34_da9d_491d_aeaf_062ce775664d.slice/crio-0af3ccd89bb8e019fabdc9651e4b8b32292a43d71f7a004bdbb2a8eb4ef4514e WatchSource:0}: Error finding container 0af3ccd89bb8e019fabdc9651e4b8b32292a43d71f7a004bdbb2a8eb4ef4514e: Status 404 returned error can't find the container with id 0af3ccd89bb8e019fabdc9651e4b8b32292a43d71f7a004bdbb2a8eb4ef4514e
Jul 02 12:49:54.643708 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:54.643691    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 12:49:55.227692 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:55.227658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j" event={"ID":"94f32a34-da9d-491d-aeaf-062ce775664d","Type":"ContainerStarted","Data":"6d4f608a3cb1b00d1254c39b8d87651372713fe4b26c49bee701b4ef83e2432d"}
Jul 02 12:49:55.227912 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:55.227710    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j" event={"ID":"94f32a34-da9d-491d-aeaf-062ce775664d","Type":"ContainerStarted","Data":"0af3ccd89bb8e019fabdc9651e4b8b32292a43d71f7a004bdbb2a8eb4ef4514e"}
Jul 02 12:49:55.245291 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:49:55.245241    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-gd68j" podStartSLOduration=1.996308687 podStartE2EDuration="2.245224971s" totalImagesPullingTime="248.916284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:49:55.244853165 +0000 UTC m=+643.228326849" watchObservedRunningTime="2026-07-02 12:49:55.245224971 +0000 UTC m=+643.228698654"
Jul 02 12:50:19.804672 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.804639    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"]
Jul 02 12:50:19.810990 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.810974    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.813526 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.813503    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rhzbx\""
Jul 02 12:50:19.813628 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.813525    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 12:50:19.813628 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.813505    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 02 12:50:19.814472 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.814454    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 12:50:19.814535 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.814484    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 02 12:50:19.814535 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.814497    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 12:50:19.814611 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.814499    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 12:50:19.819066 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.819044    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"]
Jul 02 12:50:19.891746 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.891723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b52625a-18a6-4d51-bfa6-b27d335b186f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.891872 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.891765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62v5w\" (UniqueName: \"kubernetes.io/projected/9b52625a-18a6-4d51-bfa6-b27d335b186f-kube-api-access-62v5w\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.891872 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.891793    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b52625a-18a6-4d51-bfa6-b27d335b186f-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.891872 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.891850    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b52625a-18a6-4d51-bfa6-b27d335b186f-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.891978 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.891914    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b52625a-18a6-4d51-bfa6-b27d335b186f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.992472 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.992443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62v5w\" (UniqueName: \"kubernetes.io/projected/9b52625a-18a6-4d51-bfa6-b27d335b186f-kube-api-access-62v5w\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.992608 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.992479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b52625a-18a6-4d51-bfa6-b27d335b186f-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.992608 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.992501    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b52625a-18a6-4d51-bfa6-b27d335b186f-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.992608 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.992557    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b52625a-18a6-4d51-bfa6-b27d335b186f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.992608 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.992599    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b52625a-18a6-4d51-bfa6-b27d335b186f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.993009 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.992989    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b52625a-18a6-4d51-bfa6-b27d335b186f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.993214 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.993196    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b52625a-18a6-4d51-bfa6-b27d335b186f-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.993265 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.993203    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b52625a-18a6-4d51-bfa6-b27d335b186f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:19.994945 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:19.994927    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b52625a-18a6-4d51-bfa6-b27d335b186f-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:20.001158 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:20.001126    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62v5w\" (UniqueName: \"kubernetes.io/projected/9b52625a-18a6-4d51-bfa6-b27d335b186f-kube-api-access-62v5w\") pod \"isvc-sklearn-batcher-predictor-6d976b6578-n49z7\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:20.122354 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:20.122246    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:20.256949 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:20.256927    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"]
Jul 02 12:50:20.258793 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:50:20.258766    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b52625a_18a6_4d51_bfa6_b27d335b186f.slice/crio-21c3e0460ff2159fbfee2e3cae4ab32ae19c60c03a4b0ab34dfa78179c96aa3a WatchSource:0}: Error finding container 21c3e0460ff2159fbfee2e3cae4ab32ae19c60c03a4b0ab34dfa78179c96aa3a: Status 404 returned error can't find the container with id 21c3e0460ff2159fbfee2e3cae4ab32ae19c60c03a4b0ab34dfa78179c96aa3a
Jul 02 12:50:20.339504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:20.339465    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" event={"ID":"9b52625a-18a6-4d51-bfa6-b27d335b186f","Type":"ContainerStarted","Data":"21c3e0460ff2159fbfee2e3cae4ab32ae19c60c03a4b0ab34dfa78179c96aa3a"}
Jul 02 12:50:24.361017 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:24.360984    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" event={"ID":"9b52625a-18a6-4d51-bfa6-b27d335b186f","Type":"ContainerStarted","Data":"ac0c303ec6b815969e23092ecfb37c573549cd52b780e8f7840ba32cdc4f2b11"}
Jul 02 12:50:27.376125 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:27.376090    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerID="ac0c303ec6b815969e23092ecfb37c573549cd52b780e8f7840ba32cdc4f2b11" exitCode=0
Jul 02 12:50:27.376557 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:27.376164    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" event={"ID":"9b52625a-18a6-4d51-bfa6-b27d335b186f","Type":"ContainerDied","Data":"ac0c303ec6b815969e23092ecfb37c573549cd52b780e8f7840ba32cdc4f2b11"}
Jul 02 12:50:40.451320 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:40.451268    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" event={"ID":"9b52625a-18a6-4d51-bfa6-b27d335b186f","Type":"ContainerStarted","Data":"d47864fe6d3b1e7ee5da3ad88a41f28553eb8f40a5df5feba904e7dac40c4ce2"}
Jul 02 12:50:43.469521 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:43.469482    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" event={"ID":"9b52625a-18a6-4d51-bfa6-b27d335b186f","Type":"ContainerStarted","Data":"800a6192114d3aab8e09c0be3f95f4a69c9b47685515aafead0a92cc1d0e96cd"}
Jul 02 12:50:45.481438 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:45.481402    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" event={"ID":"9b52625a-18a6-4d51-bfa6-b27d335b186f","Type":"ContainerStarted","Data":"5d3dcf831c03c247f14e738f34a1707cd87ef11e83250bdfae6174b679882ca9"}
Jul 02 12:50:45.481793 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:45.481562    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:45.507994 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:45.507950    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podStartSLOduration=2.315538822 podStartE2EDuration="26.507936708s" totalImagesPullingTime="20.520538287s" totalInitContainerRuntime="3.671859599s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:50:19 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:50:45.50575726 +0000 UTC m=+693.489230943" watchObservedRunningTime="2026-07-02 12:50:45.507936708 +0000 UTC m=+693.491410391"
Jul 02 12:50:46.486234 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:46.486203    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:46.486603 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:46.486268    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:46.487865 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:46.487827    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 12:50:46.488562 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:46.488542    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:50:47.490876 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:47.490840    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 12:50:47.491338 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:47.491178    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:50:47.495752 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:47.495731    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:50:48.495899 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:48.495858    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 12:50:48.496329 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:48.496140    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:50:58.496345 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:58.496272    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 12:50:58.496781 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:50:58.496755    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:51:08.496022 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:51:08.495939    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 12:51:08.496514 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:51:08.496487    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:51:18.496741 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:51:18.496701    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 12:51:18.497241 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:51:18.497194    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:51:28.496463 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:51:28.496413    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 12:51:28.496920 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:51:28.496900    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:51:38.496621 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:51:38.496579    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 12:51:38.497150 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:51:38.497102    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:51:48.496308 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:51:48.496266    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 12:51:48.496762 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:51:48.496738    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:51:56.422479 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:51:56.422453    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:51:56.422919 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:51:56.422527    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:52:04.950571 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.950533    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"]
Jul 02 12:52:04.951377 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.951343    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" containerID="cri-o://d47864fe6d3b1e7ee5da3ad88a41f28553eb8f40a5df5feba904e7dac40c4ce2" gracePeriod=30
Jul 02 12:52:04.951516 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.951418    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" containerID="cri-o://5d3dcf831c03c247f14e738f34a1707cd87ef11e83250bdfae6174b679882ca9" gracePeriod=30
Jul 02 12:52:04.951516 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.951493    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kube-rbac-proxy" containerID="cri-o://800a6192114d3aab8e09c0be3f95f4a69c9b47685515aafead0a92cc1d0e96cd" gracePeriod=30
Jul 02 12:52:04.961173 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.961148    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"]
Jul 02 12:52:04.965134 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.965116    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:04.967663 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.967641    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 02 12:52:04.967768 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.967660    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 02 12:52:04.997015 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.996978    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:04.997207 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.997043    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:04.997207 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.997102    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:04.997207 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.997171    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:04.997402 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.997212    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s85s\" (UniqueName: \"kubernetes.io/projected/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kube-api-access-8s85s\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:04.997662 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:04.997644    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"]
Jul 02 12:52:05.098028 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.097989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:05.098028 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.098042    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:05.098327 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.098078    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s85s\" (UniqueName: \"kubernetes.io/projected/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kube-api-access-8s85s\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:05.098327 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.098123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:05.098327 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:52:05.098154    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 02 12:52:05.098327 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.098164    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:05.098327 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:52:05.098243    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-proxy-tls podName:134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc nodeName:}" failed. No retries permitted until 2026-07-02 12:52:05.598220992 +0000 UTC m=+773.581694662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" (UID: "134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 02 12:52:05.098608 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.098585    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:05.098829 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.098811    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:05.098865 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.098833    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:05.107120 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.107100    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s85s\" (UniqueName: \"kubernetes.io/projected/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kube-api-access-8s85s\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:05.601621 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.601582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:05.604130 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.604104    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:05.847582 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.847553    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerID="800a6192114d3aab8e09c0be3f95f4a69c9b47685515aafead0a92cc1d0e96cd" exitCode=2
Jul 02 12:52:05.847711 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.847625    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" event={"ID":"9b52625a-18a6-4d51-bfa6-b27d335b186f","Type":"ContainerDied","Data":"800a6192114d3aab8e09c0be3f95f4a69c9b47685515aafead0a92cc1d0e96cd"}
Jul 02 12:52:05.875835 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:05.875760    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:06.008206 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:06.008181    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"]
Jul 02 12:52:06.010584 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:52:06.010557    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134ef92f_fdd0_4bfa_bcd1_f4b79d245fbc.slice/crio-81a0153f445117aabd66afadd8de0900d5f741925e9c1b2dc0b727a289310289 WatchSource:0}: Error finding container 81a0153f445117aabd66afadd8de0900d5f741925e9c1b2dc0b727a289310289: Status 404 returned error can't find the container with id 81a0153f445117aabd66afadd8de0900d5f741925e9c1b2dc0b727a289310289
Jul 02 12:52:06.417532 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:06.417486    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 12:52:06.417904 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:06.417880    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:52:06.853268 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:06.853222    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" event={"ID":"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc","Type":"ContainerStarted","Data":"2df22ee3ff5e2f3da9553d5b708960b5a80b52a171637de1681dd7badbc42eb3"}
Jul 02 12:52:06.853451 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:06.853338    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" event={"ID":"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc","Type":"ContainerStarted","Data":"81a0153f445117aabd66afadd8de0900d5f741925e9c1b2dc0b727a289310289"}
Jul 02 12:52:07.491587 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:07.491545    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 02 12:52:09.873361 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:09.873329    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerID="d47864fe6d3b1e7ee5da3ad88a41f28553eb8f40a5df5feba904e7dac40c4ce2" exitCode=0
Jul 02 12:52:09.873735 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:09.873413    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" event={"ID":"9b52625a-18a6-4d51-bfa6-b27d335b186f","Type":"ContainerDied","Data":"d47864fe6d3b1e7ee5da3ad88a41f28553eb8f40a5df5feba904e7dac40c4ce2"}
Jul 02 12:52:10.879504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:10.879471    2564 generic.go:356] "Generic (PLEG): container finished" podID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerID="2df22ee3ff5e2f3da9553d5b708960b5a80b52a171637de1681dd7badbc42eb3" exitCode=0
Jul 02 12:52:10.879876 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:10.879544    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" event={"ID":"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc","Type":"ContainerDied","Data":"2df22ee3ff5e2f3da9553d5b708960b5a80b52a171637de1681dd7badbc42eb3"}
Jul 02 12:52:11.886821 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:11.886791    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" event={"ID":"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc","Type":"ContainerStarted","Data":"c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c"}
Jul 02 12:52:11.887226 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:11.886852    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" event={"ID":"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc","Type":"ContainerStarted","Data":"37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280"}
Jul 02 12:52:11.887226 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:11.886879    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" event={"ID":"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc","Type":"ContainerStarted","Data":"81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395"}
Jul 02 12:52:11.887395 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:11.887253    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:11.887395 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:11.887338    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:11.887395 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:11.887376    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:11.888805 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:11.888776    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Jul 02 12:52:11.889417 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:11.889396    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:52:11.911636 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:11.911595    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podStartSLOduration=4.117219785 podStartE2EDuration="7.911583513s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.794363728s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:52:11.910277556 +0000 UTC m=+779.893751238" watchObservedRunningTime="2026-07-02 12:52:11.911583513 +0000 UTC m=+779.895057300"
Jul 02 12:52:12.491273 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:12.491231    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 02 12:52:12.891318 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:12.891204    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Jul 02 12:52:12.891799 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:12.891777    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:52:16.417629 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:16.417578    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 12:52:16.418110 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:16.417995    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:52:17.491521 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:17.491483    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 02 12:52:17.491997 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:17.491615    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:52:17.895081 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:17.895000    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:52:17.895543 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:17.895515    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Jul 02 12:52:17.895799 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:17.895775    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:52:22.491172 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:22.491127    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 02 12:52:26.417757 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:26.417708    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 12:52:26.418141 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:26.418012    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:52:26.422998 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:26.422976    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:52:26.423095 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:26.423073    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:52:27.491482 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:27.491445    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 02 12:52:27.895670 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:27.895573    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Jul 02 12:52:27.896010 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:27.895985    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:52:32.491493 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:32.491445    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 02 12:52:34.993971 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:34.993419    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerID="5d3dcf831c03c247f14e738f34a1707cd87ef11e83250bdfae6174b679882ca9" exitCode=0
Jul 02 12:52:34.993971 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:34.993574    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" event={"ID":"9b52625a-18a6-4d51-bfa6-b27d335b186f","Type":"ContainerDied","Data":"5d3dcf831c03c247f14e738f34a1707cd87ef11e83250bdfae6174b679882ca9"}
Jul 02 12:52:35.110127 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.110103    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:52:35.263846 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.263757    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b52625a-18a6-4d51-bfa6-b27d335b186f-cabundle-cert\") pod \"9b52625a-18a6-4d51-bfa6-b27d335b186f\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") "
Jul 02 12:52:35.263846 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.263834    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b52625a-18a6-4d51-bfa6-b27d335b186f-kube-rbac-proxy-sar-config\") pod \"9b52625a-18a6-4d51-bfa6-b27d335b186f\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") "
Jul 02 12:52:35.264063 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.263886    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62v5w\" (UniqueName: \"kubernetes.io/projected/9b52625a-18a6-4d51-bfa6-b27d335b186f-kube-api-access-62v5w\") pod \"9b52625a-18a6-4d51-bfa6-b27d335b186f\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") "
Jul 02 12:52:35.264063 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.263915    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b52625a-18a6-4d51-bfa6-b27d335b186f-proxy-tls\") pod \"9b52625a-18a6-4d51-bfa6-b27d335b186f\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") "
Jul 02 12:52:35.264063 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.263972    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b52625a-18a6-4d51-bfa6-b27d335b186f-kserve-provision-location\") pod \"9b52625a-18a6-4d51-bfa6-b27d335b186f\" (UID: \"9b52625a-18a6-4d51-bfa6-b27d335b186f\") "
Jul 02 12:52:35.264203 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.264152    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b52625a-18a6-4d51-bfa6-b27d335b186f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b52625a-18a6-4d51-bfa6-b27d335b186f" (UID: "9b52625a-18a6-4d51-bfa6-b27d335b186f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:52:35.264256 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.264202    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b52625a-18a6-4d51-bfa6-b27d335b186f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b52625a-18a6-4d51-bfa6-b27d335b186f" (UID: "9b52625a-18a6-4d51-bfa6-b27d335b186f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:52:35.264326 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.264261    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b52625a-18a6-4d51-bfa6-b27d335b186f-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:52:35.264400 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.264377    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b52625a-18a6-4d51-bfa6-b27d335b186f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b52625a-18a6-4d51-bfa6-b27d335b186f" (UID: "9b52625a-18a6-4d51-bfa6-b27d335b186f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:52:35.266161 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.266135    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b52625a-18a6-4d51-bfa6-b27d335b186f-kube-api-access-62v5w" (OuterVolumeSpecName: "kube-api-access-62v5w") pod "9b52625a-18a6-4d51-bfa6-b27d335b186f" (UID: "9b52625a-18a6-4d51-bfa6-b27d335b186f"). InnerVolumeSpecName "kube-api-access-62v5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:52:35.266161 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.266147    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b52625a-18a6-4d51-bfa6-b27d335b186f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b52625a-18a6-4d51-bfa6-b27d335b186f" (UID: "9b52625a-18a6-4d51-bfa6-b27d335b186f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:52:35.365101 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.365057    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b52625a-18a6-4d51-bfa6-b27d335b186f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:52:35.365101 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.365097    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62v5w\" (UniqueName: \"kubernetes.io/projected/9b52625a-18a6-4d51-bfa6-b27d335b186f-kube-api-access-62v5w\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:52:35.365286 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.365110    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b52625a-18a6-4d51-bfa6-b27d335b186f-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:52:35.365286 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.365123    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b52625a-18a6-4d51-bfa6-b27d335b186f-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:52:35.999948 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.999921    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"
Jul 02 12:52:36.000379 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:35.999922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7" event={"ID":"9b52625a-18a6-4d51-bfa6-b27d335b186f","Type":"ContainerDied","Data":"21c3e0460ff2159fbfee2e3cae4ab32ae19c60c03a4b0ab34dfa78179c96aa3a"}
Jul 02 12:52:36.000379 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:36.000062    2564 scope.go:117] "RemoveContainer" containerID="5d3dcf831c03c247f14e738f34a1707cd87ef11e83250bdfae6174b679882ca9"
Jul 02 12:52:36.008834 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:36.008819    2564 scope.go:117] "RemoveContainer" containerID="800a6192114d3aab8e09c0be3f95f4a69c9b47685515aafead0a92cc1d0e96cd"
Jul 02 12:52:36.016138 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:36.016120    2564 scope.go:117] "RemoveContainer" containerID="d47864fe6d3b1e7ee5da3ad88a41f28553eb8f40a5df5feba904e7dac40c4ce2"
Jul 02 12:52:36.023129 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:36.023104    2564 scope.go:117] "RemoveContainer" containerID="ac0c303ec6b815969e23092ecfb37c573549cd52b780e8f7840ba32cdc4f2b11"
Jul 02 12:52:36.023222 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:36.023187    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"]
Jul 02 12:52:36.027213 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:36.027194    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d976b6578-n49z7"]
Jul 02 12:52:36.422728 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:36.422656    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" path="/var/lib/kubelet/pods/9b52625a-18a6-4d51-bfa6-b27d335b186f/volumes"
Jul 02 12:52:37.895927 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:37.895889    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Jul 02 12:52:37.896402 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:37.896328    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:52:47.895487 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:47.895446    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Jul 02 12:52:47.896013 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:47.895869    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:52:57.895795 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:57.895748    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Jul 02 12:52:57.896214 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:52:57.896136    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:53:07.896050 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:07.895997    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Jul 02 12:53:07.896498 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:07.896471    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:53:17.896069 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:17.896027    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Jul 02 12:53:17.896520 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:17.896497    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:53:27.895931 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:27.895889    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:53:27.896548 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:27.896006    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:53:50.228884 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.228849    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"]
Jul 02 12:53:50.229260 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.229188    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="storage-initializer"
Jul 02 12:53:50.229260 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.229199    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="storage-initializer"
Jul 02 12:53:50.229260 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.229213    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container"
Jul 02 12:53:50.229260 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.229219    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container"
Jul 02 12:53:50.229260 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.229233    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kube-rbac-proxy"
Jul 02 12:53:50.229260 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.229240    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kube-rbac-proxy"
Jul 02 12:53:50.229260 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.229247    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent"
Jul 02 12:53:50.229260 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.229252    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent"
Jul 02 12:53:50.229533 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.229335    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kube-rbac-proxy"
Jul 02 12:53:50.229533 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.229342    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="agent"
Jul 02 12:53:50.229533 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.229349    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b52625a-18a6-4d51-bfa6-b27d335b186f" containerName="kserve-container"
Jul 02 12:53:50.232430 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.232414    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:50.235420 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.235394    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 02 12:53:50.236321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.236286    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 02 12:53:50.250884 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.250860    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"]
Jul 02 12:53:50.269789 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.269765    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"]
Jul 02 12:53:50.270056 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.270033    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" containerID="cri-o://81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395" gracePeriod=30
Jul 02 12:53:50.270143 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.270102    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" containerID="cri-o://c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c" gracePeriod=30
Jul 02 12:53:50.270203 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.270145    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kube-rbac-proxy" containerID="cri-o://37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280" gracePeriod=30
Jul 02 12:53:50.279155 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.279127    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed6be1f4-198d-4b49-915a-364d0f852567-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-dvgt5\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:50.279283 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.279172    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfv8q\" (UniqueName: \"kubernetes.io/projected/ed6be1f4-198d-4b49-915a-364d0f852567-kube-api-access-mfv8q\") pod \"message-dumper-predictor-6575bcfdfd-dvgt5\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:50.279283 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.279216    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed6be1f4-198d-4b49-915a-364d0f852567-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-dvgt5\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:50.380634 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.380606    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed6be1f4-198d-4b49-915a-364d0f852567-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-dvgt5\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:50.380787 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.380685    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed6be1f4-198d-4b49-915a-364d0f852567-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-dvgt5\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:50.380787 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.380707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfv8q\" (UniqueName: \"kubernetes.io/projected/ed6be1f4-198d-4b49-915a-364d0f852567-kube-api-access-mfv8q\") pod \"message-dumper-predictor-6575bcfdfd-dvgt5\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:50.380917 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:53:50.380809    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 02 12:53:50.380917 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:53:50.380877    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed6be1f4-198d-4b49-915a-364d0f852567-proxy-tls podName:ed6be1f4-198d-4b49-915a-364d0f852567 nodeName:}" failed. No retries permitted until 2026-07-02 12:53:50.880857692 +0000 UTC m=+878.864331359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ed6be1f4-198d-4b49-915a-364d0f852567-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-dvgt5" (UID: "ed6be1f4-198d-4b49-915a-364d0f852567") : secret "message-dumper-predictor-serving-cert" not found
Jul 02 12:53:50.381209 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.381189    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed6be1f4-198d-4b49-915a-364d0f852567-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-dvgt5\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:50.390245 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.390225    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfv8q\" (UniqueName: \"kubernetes.io/projected/ed6be1f4-198d-4b49-915a-364d0f852567-kube-api-access-mfv8q\") pod \"message-dumper-predictor-6575bcfdfd-dvgt5\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:50.884940 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.884902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed6be1f4-198d-4b49-915a-364d0f852567-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-dvgt5\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:50.887373 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:50.887347    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed6be1f4-198d-4b49-915a-364d0f852567-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-dvgt5\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:51.146766 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:51.146678    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:51.280948 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:51.280919    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"]
Jul 02 12:53:51.282883 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:53:51.282857    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6be1f4_198d_4b49_915a_364d0f852567.slice/crio-3de957ffa16ea947fb4a9d82594f852ff8b7bf43e0044557b04fd5cf69ef815b WatchSource:0}: Error finding container 3de957ffa16ea947fb4a9d82594f852ff8b7bf43e0044557b04fd5cf69ef815b: Status 404 returned error can't find the container with id 3de957ffa16ea947fb4a9d82594f852ff8b7bf43e0044557b04fd5cf69ef815b
Jul 02 12:53:51.344637 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:51.344605    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5" event={"ID":"ed6be1f4-198d-4b49-915a-364d0f852567","Type":"ContainerStarted","Data":"3de957ffa16ea947fb4a9d82594f852ff8b7bf43e0044557b04fd5cf69ef815b"}
Jul 02 12:53:51.347006 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:51.346983    2564 generic.go:356] "Generic (PLEG): container finished" podID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerID="37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280" exitCode=2
Jul 02 12:53:51.347096 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:51.347051    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" event={"ID":"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc","Type":"ContainerDied","Data":"37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280"}
Jul 02 12:53:52.891618 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:52.891580    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 02 12:53:53.358190 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:53.358157    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5" event={"ID":"ed6be1f4-198d-4b49-915a-364d0f852567","Type":"ContainerStarted","Data":"970693e480f5555c7ecd85d8642cc34163b81b3ed9d7d56318ef9c69f41dcd97"}
Jul 02 12:53:53.358398 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:53.358215    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5" event={"ID":"ed6be1f4-198d-4b49-915a-364d0f852567","Type":"ContainerStarted","Data":"3c5d686ae316ddd451fe7b70a69bdd68dc40c2061c16cb37ce49497b65d1091a"}
Jul 02 12:53:53.358398 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:53.358240    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:53.358398 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:53.358384    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:53.360104 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:53.360086    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:53:53.380037 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:53.379993    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5" podStartSLOduration=2.055476907 podStartE2EDuration="3.379979426s" totalImagesPullingTime="1.324502519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:53:53.377759089 +0000 UTC m=+881.361232772" watchObservedRunningTime="2026-07-02 12:53:53.379979426 +0000 UTC m=+881.363453109"
Jul 02 12:53:55.369689 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:55.369657    2564 generic.go:356] "Generic (PLEG): container finished" podID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerID="81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395" exitCode=0
Jul 02 12:53:55.370053 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:55.369728    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" event={"ID":"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc","Type":"ContainerDied","Data":"81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395"}
Jul 02 12:53:57.892034 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:57.891998    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 02 12:53:57.896283 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:57.896255    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Jul 02 12:53:57.896680 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:53:57.896660    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:54:00.374881 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:00.374852    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:54:02.892050 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:02.892012    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 02 12:54:02.892439 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:02.892143    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:54:07.891889 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:07.891773    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 02 12:54:07.895429 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:07.895395    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Jul 02 12:54:07.895769 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:07.895744    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:54:10.243017 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.242980    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"]
Jul 02 12:54:10.246701 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.246683    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.249497 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.249470    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 02 12:54:10.249727 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.249713    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 02 12:54:10.262919 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.262892    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"]
Jul 02 12:54:10.354209 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.354159    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c52909b-7659-4633-bc89-3bbf0919ed3f-kserve-provision-location\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.354209 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.354217    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c52909b-7659-4633-bc89-3bbf0919ed3f-cabundle-cert\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.354496 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.354263    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c52909b-7659-4633-bc89-3bbf0919ed3f-proxy-tls\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.354496 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.354288    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7hgl\" (UniqueName: \"kubernetes.io/projected/8c52909b-7659-4633-bc89-3bbf0919ed3f-kube-api-access-b7hgl\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.354496 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.354357    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c52909b-7659-4633-bc89-3bbf0919ed3f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.455389 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.455355    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c52909b-7659-4633-bc89-3bbf0919ed3f-cabundle-cert\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.455389 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.455387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c52909b-7659-4633-bc89-3bbf0919ed3f-proxy-tls\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.455629 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.455404    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7hgl\" (UniqueName: \"kubernetes.io/projected/8c52909b-7659-4633-bc89-3bbf0919ed3f-kube-api-access-b7hgl\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.455629 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.455433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c52909b-7659-4633-bc89-3bbf0919ed3f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.455629 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.455517    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c52909b-7659-4633-bc89-3bbf0919ed3f-kserve-provision-location\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.455920 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.455897    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c52909b-7659-4633-bc89-3bbf0919ed3f-kserve-provision-location\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.456121 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.456103    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c52909b-7659-4633-bc89-3bbf0919ed3f-cabundle-cert\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.456182 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.456123    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c52909b-7659-4633-bc89-3bbf0919ed3f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.457864 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.457842    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c52909b-7659-4633-bc89-3bbf0919ed3f-proxy-tls\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.464886 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.464867    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7hgl\" (UniqueName: \"kubernetes.io/projected/8c52909b-7659-4633-bc89-3bbf0919ed3f-kube-api-access-b7hgl\") pod \"isvc-logger-predictor-f76679469-6lqxv\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.556843 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.556765    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:10.706870 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:10.706849    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"]
Jul 02 12:54:10.708897 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:54:10.708858    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c52909b_7659_4633_bc89_3bbf0919ed3f.slice/crio-7cc3deab5369a05e662143a1464e22599cfb7667813c5a326611c25ae2527695 WatchSource:0}: Error finding container 7cc3deab5369a05e662143a1464e22599cfb7667813c5a326611c25ae2527695: Status 404 returned error can't find the container with id 7cc3deab5369a05e662143a1464e22599cfb7667813c5a326611c25ae2527695
Jul 02 12:54:11.439290 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:11.439255    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" event={"ID":"8c52909b-7659-4633-bc89-3bbf0919ed3f","Type":"ContainerStarted","Data":"d4a8884a00e26bf22a04a25daec63616dae6109b7ce3d3a3bdd284840146067d"}
Jul 02 12:54:11.439658 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:11.439332    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" event={"ID":"8c52909b-7659-4633-bc89-3bbf0919ed3f","Type":"ContainerStarted","Data":"7cc3deab5369a05e662143a1464e22599cfb7667813c5a326611c25ae2527695"}
Jul 02 12:54:12.891891 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:12.891848    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 02 12:54:15.460231 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:15.460195    2564 generic.go:356] "Generic (PLEG): container finished" podID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerID="d4a8884a00e26bf22a04a25daec63616dae6109b7ce3d3a3bdd284840146067d" exitCode=0
Jul 02 12:54:15.460635 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:15.460272    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" event={"ID":"8c52909b-7659-4633-bc89-3bbf0919ed3f","Type":"ContainerDied","Data":"d4a8884a00e26bf22a04a25daec63616dae6109b7ce3d3a3bdd284840146067d"}
Jul 02 12:54:16.468118 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:16.468084    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" event={"ID":"8c52909b-7659-4633-bc89-3bbf0919ed3f","Type":"ContainerStarted","Data":"5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48"}
Jul 02 12:54:16.468528 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:16.468148    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" event={"ID":"8c52909b-7659-4633-bc89-3bbf0919ed3f","Type":"ContainerStarted","Data":"67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36"}
Jul 02 12:54:16.468528 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:16.468186    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" event={"ID":"8c52909b-7659-4633-bc89-3bbf0919ed3f","Type":"ContainerStarted","Data":"87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f"}
Jul 02 12:54:16.468528 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:16.468463    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:16.468688 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:16.468599    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:16.470085 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:16.470058    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:54:16.500354 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:16.500313    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podStartSLOduration=2.623998564 podStartE2EDuration="6.500289153s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.876290589s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:54:16.497668691 +0000 UTC m=+904.481142369" watchObservedRunningTime="2026-07-02 12:54:16.500289153 +0000 UTC m=+904.483762835"
Jul 02 12:54:17.473889 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:17.473856    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:17.474374 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:17.473988    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:54:17.475213 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:17.475185    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:54:17.891509 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:17.891421    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 02 12:54:17.895733 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:17.895698    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Jul 02 12:54:17.895878 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:17.895818    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:54:17.896122 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:17.896104    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:54:17.896202 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:17.896189    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:54:18.478365 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:18.478329    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:54:18.478771 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:18.478723    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:54:20.427852 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.427829    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:54:20.488337 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.488284    2564 generic.go:356] "Generic (PLEG): container finished" podID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerID="c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c" exitCode=0
Jul 02 12:54:20.488337 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.488334    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" event={"ID":"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc","Type":"ContainerDied","Data":"c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c"}
Jul 02 12:54:20.488509 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.488405    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8" event={"ID":"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc","Type":"ContainerDied","Data":"81a0153f445117aabd66afadd8de0900d5f741925e9c1b2dc0b727a289310289"}
Jul 02 12:54:20.488509 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.488419    2564 scope.go:117] "RemoveContainer" containerID="c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c"
Jul 02 12:54:20.488509 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.488405    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"
Jul 02 12:54:20.496408 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.496392    2564 scope.go:117] "RemoveContainer" containerID="37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280"
Jul 02 12:54:20.503489 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.503473    2564 scope.go:117] "RemoveContainer" containerID="81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395"
Jul 02 12:54:20.510356 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.510342    2564 scope.go:117] "RemoveContainer" containerID="2df22ee3ff5e2f3da9553d5b708960b5a80b52a171637de1681dd7badbc42eb3"
Jul 02 12:54:20.517053 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.517036    2564 scope.go:117] "RemoveContainer" containerID="c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c"
Jul 02 12:54:20.517314 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:54:20.517275    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c\": container with ID starting with c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c not found: ID does not exist" containerID="c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c"
Jul 02 12:54:20.517418 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.517318    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c"} err="failed to get container status \"c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c\": rpc error: code = NotFound desc = could not find container \"c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c\": container with ID starting with c29af2ab986116a77797a01a80024d4b2d407404b5c6d2892a42f30c2cbcb40c not found: ID does not exist"
Jul 02 12:54:20.517418 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.517341    2564 scope.go:117] "RemoveContainer" containerID="37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280"
Jul 02 12:54:20.517618 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:54:20.517598    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280\": container with ID starting with 37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280 not found: ID does not exist" containerID="37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280"
Jul 02 12:54:20.517654 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.517623    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280"} err="failed to get container status \"37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280\": rpc error: code = NotFound desc = could not find container \"37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280\": container with ID starting with 37492e98ce16d97cf0aa8d6388e9bc0ea34bdbf25366d0a0a96342b4a94dc280 not found: ID does not exist"
Jul 02 12:54:20.517654 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.517638    2564 scope.go:117] "RemoveContainer" containerID="81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395"
Jul 02 12:54:20.517862 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:54:20.517844    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395\": container with ID starting with 81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395 not found: ID does not exist" containerID="81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395"
Jul 02 12:54:20.517909 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.517867    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395"} err="failed to get container status \"81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395\": rpc error: code = NotFound desc = could not find container \"81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395\": container with ID starting with 81a333b16e342c8e27677b6b62d8416458b99b3692a5b2fd0424f71f796ef395 not found: ID does not exist"
Jul 02 12:54:20.517909 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.517881    2564 scope.go:117] "RemoveContainer" containerID="2df22ee3ff5e2f3da9553d5b708960b5a80b52a171637de1681dd7badbc42eb3"
Jul 02 12:54:20.518106 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:54:20.518088    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df22ee3ff5e2f3da9553d5b708960b5a80b52a171637de1681dd7badbc42eb3\": container with ID starting with 2df22ee3ff5e2f3da9553d5b708960b5a80b52a171637de1681dd7badbc42eb3 not found: ID does not exist" containerID="2df22ee3ff5e2f3da9553d5b708960b5a80b52a171637de1681dd7badbc42eb3"
Jul 02 12:54:20.518157 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.518113    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df22ee3ff5e2f3da9553d5b708960b5a80b52a171637de1681dd7badbc42eb3"} err="failed to get container status \"2df22ee3ff5e2f3da9553d5b708960b5a80b52a171637de1681dd7badbc42eb3\": rpc error: code = NotFound desc = could not find container \"2df22ee3ff5e2f3da9553d5b708960b5a80b52a171637de1681dd7badbc42eb3\": container with ID starting with 2df22ee3ff5e2f3da9553d5b708960b5a80b52a171637de1681dd7badbc42eb3 not found: ID does not exist"
Jul 02 12:54:20.544466 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.544421    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kube-rbac-proxy-sar-config\") pod \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") "
Jul 02 12:54:20.544466 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.544448    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s85s\" (UniqueName: \"kubernetes.io/projected/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kube-api-access-8s85s\") pod \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") "
Jul 02 12:54:20.544585 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.544504    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-proxy-tls\") pod \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") "
Jul 02 12:54:20.544585 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.544526    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kserve-provision-location\") pod \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") "
Jul 02 12:54:20.544585 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.544555    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-cabundle-cert\") pod \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\" (UID: \"134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc\") "
Jul 02 12:54:20.544882 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.544853    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" (UID: "134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:54:20.544963 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.544884    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" (UID: "134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:54:20.545008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.544983    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" (UID: "134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:54:20.546456 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.546438    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" (UID: "134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:54:20.546620 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.546601    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kube-api-access-8s85s" (OuterVolumeSpecName: "kube-api-access-8s85s") pod "134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" (UID: "134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc"). InnerVolumeSpecName "kube-api-access-8s85s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:54:20.645252 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.645205    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:54:20.645252 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.645238    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s85s\" (UniqueName: \"kubernetes.io/projected/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kube-api-access-8s85s\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:54:20.645252 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.645249    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:54:20.645252 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.645257    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:54:20.645581 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.645266    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:54:20.812459 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.812425    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"]
Jul 02 12:54:20.816166 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:20.816141    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7cf959d945-bkbx8"]
Jul 02 12:54:22.422786 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:22.422752    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" path="/var/lib/kubelet/pods/134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc/volumes"
Jul 02 12:54:23.482419 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:23.482391    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:54:23.482978 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:23.482949    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:54:23.483312 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:23.483262    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:54:33.483148 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:33.483106    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:54:33.483711 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:33.483631    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:54:43.483279 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:43.483239    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:54:43.483732 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:43.483706    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:54:53.483113 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:53.483065    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:54:53.485438 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:54:53.483573    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:55:03.482927 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:03.482888    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:55:03.483457 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:03.483428    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:55:13.482841 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:13.482801    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:55:13.483338 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:13.483311    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:55:23.482882 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:23.482835    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:55:23.483368 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:23.483278    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:55:33.418404 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:33.418372    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:55:33.418915 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:33.418896    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:55:45.284529 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.284436    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-dvgt5_ed6be1f4-198d-4b49-915a-364d0f852567/kserve-container/0.log"
Jul 02 12:55:45.635872 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.635796    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"]
Jul 02 12:55:45.636105 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.636073    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" containerID="cri-o://5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48" gracePeriod=30
Jul 02 12:55:45.636255 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.636137    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kube-rbac-proxy" containerID="cri-o://67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36" gracePeriod=30
Jul 02 12:55:45.636255 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.636058    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" containerID="cri-o://87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f" gracePeriod=30
Jul 02 12:55:45.643897 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.643865    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"]
Jul 02 12:55:45.644212 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.644200    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent"
Jul 02 12:55:45.644271 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.644214    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent"
Jul 02 12:55:45.644271 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.644231    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="storage-initializer"
Jul 02 12:55:45.644271 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.644237    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="storage-initializer"
Jul 02 12:55:45.644271 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.644245    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kube-rbac-proxy"
Jul 02 12:55:45.644271 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.644250    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kube-rbac-proxy"
Jul 02 12:55:45.644271 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.644260    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container"
Jul 02 12:55:45.644271 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.644265    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container"
Jul 02 12:55:45.644618 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.644336    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kserve-container"
Jul 02 12:55:45.644618 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.644344    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="kube-rbac-proxy"
Jul 02 12:55:45.644618 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.644350    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="134ef92f-fdd0-4bfa-bcd1-f4b79d245fbc" containerName="agent"
Jul 02 12:55:45.647624 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.647605    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.651178 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.651161    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 02 12:55:45.654110 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.654093    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 02 12:55:45.665116 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.665095    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"]
Jul 02 12:55:45.701236 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.701186    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"]
Jul 02 12:55:45.701616 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.701583    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5" podUID="ed6be1f4-198d-4b49-915a-364d0f852567" containerName="kserve-container" containerID="cri-o://3c5d686ae316ddd451fe7b70a69bdd68dc40c2061c16cb37ce49497b65d1091a" gracePeriod=30
Jul 02 12:55:45.701769 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.701708    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5" podUID="ed6be1f4-198d-4b49-915a-364d0f852567" containerName="kube-rbac-proxy" containerID="cri-o://970693e480f5555c7ecd85d8642cc34163b81b3ed9d7d56318ef9c69f41dcd97" gracePeriod=30
Jul 02 12:55:45.762949 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.762927    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.763066 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.762969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcfdv\" (UniqueName: \"kubernetes.io/projected/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kube-api-access-wcfdv\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.763066 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.763031    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df5f6ed-7cde-4590-b7d2-bd52620d608d-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.763159 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.763128    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df5f6ed-7cde-4590-b7d2-bd52620d608d-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.763214 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.763193    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.864112 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.864077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df5f6ed-7cde-4590-b7d2-bd52620d608d-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.864275 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.864126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.864275 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.864167    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.864275 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.864192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcfdv\" (UniqueName: \"kubernetes.io/projected/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kube-api-access-wcfdv\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.864275 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.864224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df5f6ed-7cde-4590-b7d2-bd52620d608d-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.864275 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:55:45.864246    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 02 12:55:45.864674 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:55:45.864648    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df5f6ed-7cde-4590-b7d2-bd52620d608d-proxy-tls podName:4df5f6ed-7cde-4590-b7d2-bd52620d608d nodeName:}" failed. No retries permitted until 2026-07-02 12:55:46.364620978 +0000 UTC m=+994.348094652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4df5f6ed-7cde-4590-b7d2-bd52620d608d-proxy-tls") pod "isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" (UID: "4df5f6ed-7cde-4590-b7d2-bd52620d608d") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 02 12:55:45.864806 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.864727    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.864928 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.864909    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.864992 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.864943    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df5f6ed-7cde-4590-b7d2-bd52620d608d-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.877128 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.877103    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcfdv\" (UniqueName: \"kubernetes.io/projected/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kube-api-access-wcfdv\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:45.884147 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.884123    2564 generic.go:356] "Generic (PLEG): container finished" podID="ed6be1f4-198d-4b49-915a-364d0f852567" containerID="970693e480f5555c7ecd85d8642cc34163b81b3ed9d7d56318ef9c69f41dcd97" exitCode=2
Jul 02 12:55:45.884147 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.884144    2564 generic.go:356] "Generic (PLEG): container finished" podID="ed6be1f4-198d-4b49-915a-364d0f852567" containerID="3c5d686ae316ddd451fe7b70a69bdd68dc40c2061c16cb37ce49497b65d1091a" exitCode=2
Jul 02 12:55:45.884319 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.884201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5" event={"ID":"ed6be1f4-198d-4b49-915a-364d0f852567","Type":"ContainerDied","Data":"970693e480f5555c7ecd85d8642cc34163b81b3ed9d7d56318ef9c69f41dcd97"}
Jul 02 12:55:45.884319 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.884266    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5" event={"ID":"ed6be1f4-198d-4b49-915a-364d0f852567","Type":"ContainerDied","Data":"3c5d686ae316ddd451fe7b70a69bdd68dc40c2061c16cb37ce49497b65d1091a"}
Jul 02 12:55:45.887321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.887242    2564 generic.go:356] "Generic (PLEG): container finished" podID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerID="67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36" exitCode=2
Jul 02 12:55:45.887419 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.887358    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" event={"ID":"8c52909b-7659-4633-bc89-3bbf0919ed3f","Type":"ContainerDied","Data":"67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36"}
Jul 02 12:55:45.952286 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:45.952265    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:55:46.066306 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.066278    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed6be1f4-198d-4b49-915a-364d0f852567-proxy-tls\") pod \"ed6be1f4-198d-4b49-915a-364d0f852567\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") "
Jul 02 12:55:46.066459 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.066383    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfv8q\" (UniqueName: \"kubernetes.io/projected/ed6be1f4-198d-4b49-915a-364d0f852567-kube-api-access-mfv8q\") pod \"ed6be1f4-198d-4b49-915a-364d0f852567\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") "
Jul 02 12:55:46.066459 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.066426    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed6be1f4-198d-4b49-915a-364d0f852567-kube-rbac-proxy-sar-config\") pod \"ed6be1f4-198d-4b49-915a-364d0f852567\" (UID: \"ed6be1f4-198d-4b49-915a-364d0f852567\") "
Jul 02 12:55:46.066808 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.066776    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6be1f4-198d-4b49-915a-364d0f852567-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed6be1f4-198d-4b49-915a-364d0f852567" (UID: "ed6be1f4-198d-4b49-915a-364d0f852567"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:55:46.068440 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.068417    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed6be1f4-198d-4b49-915a-364d0f852567-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed6be1f4-198d-4b49-915a-364d0f852567" (UID: "ed6be1f4-198d-4b49-915a-364d0f852567"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:55:46.068543 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.068518    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6be1f4-198d-4b49-915a-364d0f852567-kube-api-access-mfv8q" (OuterVolumeSpecName: "kube-api-access-mfv8q") pod "ed6be1f4-198d-4b49-915a-364d0f852567" (UID: "ed6be1f4-198d-4b49-915a-364d0f852567"). InnerVolumeSpecName "kube-api-access-mfv8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:55:46.168024 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.167958    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed6be1f4-198d-4b49-915a-364d0f852567-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:55:46.168024 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.167982    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed6be1f4-198d-4b49-915a-364d0f852567-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:55:46.168024 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.167991    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfv8q\" (UniqueName: \"kubernetes.io/projected/ed6be1f4-198d-4b49-915a-364d0f852567-kube-api-access-mfv8q\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:55:46.369729 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.369694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df5f6ed-7cde-4590-b7d2-bd52620d608d-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:46.372094 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.372073    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df5f6ed-7cde-4590-b7d2-bd52620d608d-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:46.558916 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.558882    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:55:46.700859 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.700832    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"]
Jul 02 12:55:46.702873 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:55:46.702846    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df5f6ed_7cde_4590_b7d2_bd52620d608d.slice/crio-96bdc4c806e6cdb2fe50b02d928ca3c3e854eba8d35bedaaecdc5d1ebebc9f23 WatchSource:0}: Error finding container 96bdc4c806e6cdb2fe50b02d928ca3c3e854eba8d35bedaaecdc5d1ebebc9f23: Status 404 returned error can't find the container with id 96bdc4c806e6cdb2fe50b02d928ca3c3e854eba8d35bedaaecdc5d1ebebc9f23
Jul 02 12:55:46.894315 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.894264    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" event={"ID":"4df5f6ed-7cde-4590-b7d2-bd52620d608d","Type":"ContainerStarted","Data":"d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8"}
Jul 02 12:55:46.894503 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.894346    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" event={"ID":"4df5f6ed-7cde-4590-b7d2-bd52620d608d","Type":"ContainerStarted","Data":"96bdc4c806e6cdb2fe50b02d928ca3c3e854eba8d35bedaaecdc5d1ebebc9f23"}
Jul 02 12:55:46.896074 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.896049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5" event={"ID":"ed6be1f4-198d-4b49-915a-364d0f852567","Type":"ContainerDied","Data":"3de957ffa16ea947fb4a9d82594f852ff8b7bf43e0044557b04fd5cf69ef815b"}
Jul 02 12:55:46.896204 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.896075    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"
Jul 02 12:55:46.896204 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.896107    2564 scope.go:117] "RemoveContainer" containerID="970693e480f5555c7ecd85d8642cc34163b81b3ed9d7d56318ef9c69f41dcd97"
Jul 02 12:55:46.906362 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.906344    2564 scope.go:117] "RemoveContainer" containerID="3c5d686ae316ddd451fe7b70a69bdd68dc40c2061c16cb37ce49497b65d1091a"
Jul 02 12:55:46.931051 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.931027    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"]
Jul 02 12:55:46.937686 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:46.937661    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dvgt5"]
Jul 02 12:55:48.428286 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:48.428247    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed6be1f4-198d-4b49-915a-364d0f852567" path="/var/lib/kubelet/pods/ed6be1f4-198d-4b49-915a-364d0f852567/volumes"
Jul 02 12:55:48.478942 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:48.478900    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 02 12:55:50.916697 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:50.916666    2564 generic.go:356] "Generic (PLEG): container finished" podID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerID="87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f" exitCode=0
Jul 02 12:55:50.917076 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:50.916726    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" event={"ID":"8c52909b-7659-4633-bc89-3bbf0919ed3f","Type":"ContainerDied","Data":"87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f"}
Jul 02 12:55:53.417850 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:53.417809    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:55:53.418227 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:53.418129    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:55:53.479335 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:53.479277    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 02 12:55:58.478822 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:58.478781    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 02 12:55:58.479233 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:55:58.478903    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:56:03.418838 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:03.418790    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:56:03.419242 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:03.419121    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:56:03.479335 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:03.479274    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 02 12:56:08.478656 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:08.478612    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 02 12:56:13.418367 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:13.418322    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:56:13.418805 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:13.418473    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:56:13.418805 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:13.418646    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:56:13.418805 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:13.418753    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:56:13.478840 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:13.478803    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 02 12:56:15.838820 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:15.838791    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:56:16.019551 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.019503    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c52909b-7659-4633-bc89-3bbf0919ed3f-kube-rbac-proxy-sar-config\") pod \"8c52909b-7659-4633-bc89-3bbf0919ed3f\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") "
Jul 02 12:56:16.019734 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.019584    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c52909b-7659-4633-bc89-3bbf0919ed3f-cabundle-cert\") pod \"8c52909b-7659-4633-bc89-3bbf0919ed3f\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") "
Jul 02 12:56:16.019734 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.019704    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c52909b-7659-4633-bc89-3bbf0919ed3f-proxy-tls\") pod \"8c52909b-7659-4633-bc89-3bbf0919ed3f\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") "
Jul 02 12:56:16.019822 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.019788    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c52909b-7659-4633-bc89-3bbf0919ed3f-kserve-provision-location\") pod \"8c52909b-7659-4633-bc89-3bbf0919ed3f\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") "
Jul 02 12:56:16.019860 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.019827    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7hgl\" (UniqueName: \"kubernetes.io/projected/8c52909b-7659-4633-bc89-3bbf0919ed3f-kube-api-access-b7hgl\") pod \"8c52909b-7659-4633-bc89-3bbf0919ed3f\" (UID: \"8c52909b-7659-4633-bc89-3bbf0919ed3f\") "
Jul 02 12:56:16.019943 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.019907    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c52909b-7659-4633-bc89-3bbf0919ed3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c52909b-7659-4633-bc89-3bbf0919ed3f" (UID: "8c52909b-7659-4633-bc89-3bbf0919ed3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:56:16.019943 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.019915    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c52909b-7659-4633-bc89-3bbf0919ed3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c52909b-7659-4633-bc89-3bbf0919ed3f" (UID: "8c52909b-7659-4633-bc89-3bbf0919ed3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:56:16.020131 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.020107    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c52909b-7659-4633-bc89-3bbf0919ed3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c52909b-7659-4633-bc89-3bbf0919ed3f" (UID: "8c52909b-7659-4633-bc89-3bbf0919ed3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:56:16.020225 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.020211    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c52909b-7659-4633-bc89-3bbf0919ed3f-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:56:16.020269 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.020228    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c52909b-7659-4633-bc89-3bbf0919ed3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:56:16.020269 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.020237    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c52909b-7659-4633-bc89-3bbf0919ed3f-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:56:16.021787 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.021766    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c52909b-7659-4633-bc89-3bbf0919ed3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c52909b-7659-4633-bc89-3bbf0919ed3f" (UID: "8c52909b-7659-4633-bc89-3bbf0919ed3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:56:16.021842 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.021810    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c52909b-7659-4633-bc89-3bbf0919ed3f-kube-api-access-b7hgl" (OuterVolumeSpecName: "kube-api-access-b7hgl") pod "8c52909b-7659-4633-bc89-3bbf0919ed3f" (UID: "8c52909b-7659-4633-bc89-3bbf0919ed3f"). InnerVolumeSpecName "kube-api-access-b7hgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:56:16.037068 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.037045    2564 generic.go:356] "Generic (PLEG): container finished" podID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerID="5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48" exitCode=0
Jul 02 12:56:16.037170 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.037080    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" event={"ID":"8c52909b-7659-4633-bc89-3bbf0919ed3f","Type":"ContainerDied","Data":"5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48"}
Jul 02 12:56:16.037170 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.037102    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"
Jul 02 12:56:16.037170 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.037131    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv" event={"ID":"8c52909b-7659-4633-bc89-3bbf0919ed3f","Type":"ContainerDied","Data":"7cc3deab5369a05e662143a1464e22599cfb7667813c5a326611c25ae2527695"}
Jul 02 12:56:16.037170 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.037162    2564 scope.go:117] "RemoveContainer" containerID="5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48"
Jul 02 12:56:16.045480 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.045466    2564 scope.go:117] "RemoveContainer" containerID="67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36"
Jul 02 12:56:16.053102 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.053081    2564 scope.go:117] "RemoveContainer" containerID="87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f"
Jul 02 12:56:16.063267 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.063252    2564 scope.go:117] "RemoveContainer" containerID="d4a8884a00e26bf22a04a25daec63616dae6109b7ce3d3a3bdd284840146067d"
Jul 02 12:56:16.064154 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.064096    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"]
Jul 02 12:56:16.067207 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.067187    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-f76679469-6lqxv"]
Jul 02 12:56:16.070428 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.070414    2564 scope.go:117] "RemoveContainer" containerID="5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48"
Jul 02 12:56:16.070684 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:56:16.070667    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48\": container with ID starting with 5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48 not found: ID does not exist" containerID="5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48"
Jul 02 12:56:16.070753 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.070697    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48"} err="failed to get container status \"5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48\": rpc error: code = NotFound desc = could not find container \"5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48\": container with ID starting with 5e098ecdbf48465e22de77cdb009dcd2d434c09438b6f9105dfed962d803da48 not found: ID does not exist"
Jul 02 12:56:16.070753 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.070715    2564 scope.go:117] "RemoveContainer" containerID="67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36"
Jul 02 12:56:16.070942 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:56:16.070924    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36\": container with ID starting with 67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36 not found: ID does not exist" containerID="67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36"
Jul 02 12:56:16.070982 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.070949    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36"} err="failed to get container status \"67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36\": rpc error: code = NotFound desc = could not find container \"67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36\": container with ID starting with 67bcdfc3981dbd9371c0eb9ab8509f310f6fc01277f0917434ca154e3e231c36 not found: ID does not exist"
Jul 02 12:56:16.070982 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.070965    2564 scope.go:117] "RemoveContainer" containerID="87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f"
Jul 02 12:56:16.071165 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:56:16.071148    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f\": container with ID starting with 87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f not found: ID does not exist" containerID="87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f"
Jul 02 12:56:16.071205 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.071170    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f"} err="failed to get container status \"87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f\": rpc error: code = NotFound desc = could not find container \"87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f\": container with ID starting with 87723a8a8df42a9fb818982104c1dd4263ba1c9f5f7bf580e371ec475aa0116f not found: ID does not exist"
Jul 02 12:56:16.071205 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.071185    2564 scope.go:117] "RemoveContainer" containerID="d4a8884a00e26bf22a04a25daec63616dae6109b7ce3d3a3bdd284840146067d"
Jul 02 12:56:16.071471 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:56:16.071451    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a8884a00e26bf22a04a25daec63616dae6109b7ce3d3a3bdd284840146067d\": container with ID starting with d4a8884a00e26bf22a04a25daec63616dae6109b7ce3d3a3bdd284840146067d not found: ID does not exist" containerID="d4a8884a00e26bf22a04a25daec63616dae6109b7ce3d3a3bdd284840146067d"
Jul 02 12:56:16.071524 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.071478    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a8884a00e26bf22a04a25daec63616dae6109b7ce3d3a3bdd284840146067d"} err="failed to get container status \"d4a8884a00e26bf22a04a25daec63616dae6109b7ce3d3a3bdd284840146067d\": rpc error: code = NotFound desc = could not find container \"d4a8884a00e26bf22a04a25daec63616dae6109b7ce3d3a3bdd284840146067d\": container with ID starting with d4a8884a00e26bf22a04a25daec63616dae6109b7ce3d3a3bdd284840146067d not found: ID does not exist"
Jul 02 12:56:16.120965 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.120935    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c52909b-7659-4633-bc89-3bbf0919ed3f-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:56:16.120965 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.120962    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7hgl\" (UniqueName: \"kubernetes.io/projected/8c52909b-7659-4633-bc89-3bbf0919ed3f-kube-api-access-b7hgl\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:56:16.423123 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:16.423081    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" path="/var/lib/kubelet/pods/8c52909b-7659-4633-bc89-3bbf0919ed3f/volumes"
Jul 02 12:56:17.043689 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:17.043658    2564 generic.go:356] "Generic (PLEG): container finished" podID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerID="d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8" exitCode=0
Jul 02 12:56:17.044106 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:17.043700    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" event={"ID":"4df5f6ed-7cde-4590-b7d2-bd52620d608d","Type":"ContainerDied","Data":"d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8"}
Jul 02 12:56:17.044996 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:56:17.044982    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 12:58:35.749962 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:35.749922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" event={"ID":"4df5f6ed-7cde-4590-b7d2-bd52620d608d","Type":"ContainerStarted","Data":"ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb"}
Jul 02 12:58:35.750487 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:35.749992    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" event={"ID":"4df5f6ed-7cde-4590-b7d2-bd52620d608d","Type":"ContainerStarted","Data":"b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977"}
Jul 02 12:58:35.750487 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:35.750141    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:58:35.750487 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:35.750188    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:58:35.751683 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:35.751660    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 02 12:58:35.781956 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:35.781911    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" podStartSLOduration=3.624204517 podStartE2EDuration="2m50.781878673s" totalImagesPullingTime="2m17.598930706s" totalInitContainerRuntime="29.55874345s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:58:35.781545029 +0000 UTC m=+1163.765018712" watchObservedRunningTime="2026-07-02 12:58:35.781878673 +0000 UTC m=+1163.765352358"
Jul 02 12:58:36.755211 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:36.755170    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 02 12:58:41.759696 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:41.759668    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:58:41.760213 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:41.760129    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 02 12:58:51.762118 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:51.762089    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:58:55.777262 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:55.777228    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"]
Jul 02 12:58:55.777703 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:55.777453    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="kserve-container" containerID="cri-o://b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977" gracePeriod=30
Jul 02 12:58:55.777703 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:55.777544    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="kube-rbac-proxy" containerID="cri-o://ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb" gracePeriod=30
Jul 02 12:58:56.656922 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.656863    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:58:56.746354 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.746324    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kserve-provision-location\") pod \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") "
Jul 02 12:58:56.746500 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.746359    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcfdv\" (UniqueName: \"kubernetes.io/projected/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kube-api-access-wcfdv\") pod \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") "
Jul 02 12:58:56.746500 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.746408    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df5f6ed-7cde-4590-b7d2-bd52620d608d-proxy-tls\") pod \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") "
Jul 02 12:58:56.746500 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.746466    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kube-rbac-proxy-sar-config\") pod \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") "
Jul 02 12:58:56.746629 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.746506    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df5f6ed-7cde-4590-b7d2-bd52620d608d-cabundle-cert\") pod \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\" (UID: \"4df5f6ed-7cde-4590-b7d2-bd52620d608d\") "
Jul 02 12:58:56.746894 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.746873    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4df5f6ed-7cde-4590-b7d2-bd52620d608d" (UID: "4df5f6ed-7cde-4590-b7d2-bd52620d608d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:58:56.746968 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.746904    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df5f6ed-7cde-4590-b7d2-bd52620d608d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4df5f6ed-7cde-4590-b7d2-bd52620d608d" (UID: "4df5f6ed-7cde-4590-b7d2-bd52620d608d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:58:56.748681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.748652    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df5f6ed-7cde-4590-b7d2-bd52620d608d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4df5f6ed-7cde-4590-b7d2-bd52620d608d" (UID: "4df5f6ed-7cde-4590-b7d2-bd52620d608d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:58:56.748681 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.748655    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kube-api-access-wcfdv" (OuterVolumeSpecName: "kube-api-access-wcfdv") pod "4df5f6ed-7cde-4590-b7d2-bd52620d608d" (UID: "4df5f6ed-7cde-4590-b7d2-bd52620d608d"). InnerVolumeSpecName "kube-api-access-wcfdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:58:56.751454 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.751430    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4df5f6ed-7cde-4590-b7d2-bd52620d608d" (UID: "4df5f6ed-7cde-4590-b7d2-bd52620d608d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:58:56.847111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.847088    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:58:56.847111 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.847110    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcfdv\" (UniqueName: \"kubernetes.io/projected/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kube-api-access-wcfdv\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:58:56.847504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.847124    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df5f6ed-7cde-4590-b7d2-bd52620d608d-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:58:56.847504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.847133    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df5f6ed-7cde-4590-b7d2-bd52620d608d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:58:56.847504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.847143    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df5f6ed-7cde-4590-b7d2-bd52620d608d-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 12:58:56.847504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.847262    2564 generic.go:356] "Generic (PLEG): container finished" podID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerID="ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb" exitCode=2
Jul 02 12:58:56.847504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.847287    2564 generic.go:356] "Generic (PLEG): container finished" podID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerID="b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977" exitCode=0
Jul 02 12:58:56.847504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.847332    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" event={"ID":"4df5f6ed-7cde-4590-b7d2-bd52620d608d","Type":"ContainerDied","Data":"ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb"}
Jul 02 12:58:56.847504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.847352    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"
Jul 02 12:58:56.847504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.847417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" event={"ID":"4df5f6ed-7cde-4590-b7d2-bd52620d608d","Type":"ContainerDied","Data":"b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977"}
Jul 02 12:58:56.847504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.847439    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v" event={"ID":"4df5f6ed-7cde-4590-b7d2-bd52620d608d","Type":"ContainerDied","Data":"96bdc4c806e6cdb2fe50b02d928ca3c3e854eba8d35bedaaecdc5d1ebebc9f23"}
Jul 02 12:58:56.847504 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.847466    2564 scope.go:117] "RemoveContainer" containerID="ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb"
Jul 02 12:58:56.856367 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.856350    2564 scope.go:117] "RemoveContainer" containerID="b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977"
Jul 02 12:58:56.864082 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.864061    2564 scope.go:117] "RemoveContainer" containerID="d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8"
Jul 02 12:58:56.870146 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.870126    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"]
Jul 02 12:58:56.874116 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.874099    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lxh5v"]
Jul 02 12:58:56.885323 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.885281    2564 scope.go:117] "RemoveContainer" containerID="ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb"
Jul 02 12:58:56.885606 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:58:56.885585    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb\": container with ID starting with ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb not found: ID does not exist" containerID="ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb"
Jul 02 12:58:56.885688 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.885614    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb"} err="failed to get container status \"ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb\": rpc error: code = NotFound desc = could not find container \"ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb\": container with ID starting with ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb not found: ID does not exist"
Jul 02 12:58:56.885688 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.885632    2564 scope.go:117] "RemoveContainer" containerID="b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977"
Jul 02 12:58:56.885866 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:58:56.885849    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977\": container with ID starting with b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977 not found: ID does not exist" containerID="b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977"
Jul 02 12:58:56.885912 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.885869    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977"} err="failed to get container status \"b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977\": rpc error: code = NotFound desc = could not find container \"b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977\": container with ID starting with b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977 not found: ID does not exist"
Jul 02 12:58:56.885912 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.885884    2564 scope.go:117] "RemoveContainer" containerID="d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8"
Jul 02 12:58:56.886133 ip-10-0-143-149 kubenswrapper[2564]: E0702 12:58:56.886112    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8\": container with ID starting with d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8 not found: ID does not exist" containerID="d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8"
Jul 02 12:58:56.886205 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.886143    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8"} err="failed to get container status \"d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8\": rpc error: code = NotFound desc = could not find container \"d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8\": container with ID starting with d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8 not found: ID does not exist"
Jul 02 12:58:56.886205 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.886165    2564 scope.go:117] "RemoveContainer" containerID="ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb"
Jul 02 12:58:56.886440 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.886420    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb"} err="failed to get container status \"ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb\": rpc error: code = NotFound desc = could not find container \"ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb\": container with ID starting with ade31c7ae20be400a6db399b32b6ab845bb4f3305800226838260e4c6a0877cb not found: ID does not exist"
Jul 02 12:58:56.886509 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.886443    2564 scope.go:117] "RemoveContainer" containerID="b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977"
Jul 02 12:58:56.886730 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.886708    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977"} err="failed to get container status \"b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977\": rpc error: code = NotFound desc = could not find container \"b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977\": container with ID starting with b9a08dd47b2993008a7cdd811671b3f2709ef0c32401d068c5fd492d1f7f6977 not found: ID does not exist"
Jul 02 12:58:56.886808 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.886732    2564 scope.go:117] "RemoveContainer" containerID="d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8"
Jul 02 12:58:56.887008 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:56.886989    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8"} err="failed to get container status \"d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8\": rpc error: code = NotFound desc = could not find container \"d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8\": container with ID starting with d7afe266d16059f9b3973f29775905c77ea06ca53580256d77da97a8be36b8b8 not found: ID does not exist"
Jul 02 12:58:58.423670 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:58:58.423635    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" path="/var/lib/kubelet/pods/4df5f6ed-7cde-4590-b7d2-bd52620d608d/volumes"
Jul 02 12:59:01.066595 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066564    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"]
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066882    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="kserve-container"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066893    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="kserve-container"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066903    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="kube-rbac-proxy"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066909    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="kube-rbac-proxy"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066921    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6be1f4-198d-4b49-915a-364d0f852567" containerName="kube-rbac-proxy"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066927    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6be1f4-198d-4b49-915a-364d0f852567" containerName="kube-rbac-proxy"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066940    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6be1f4-198d-4b49-915a-364d0f852567" containerName="kserve-container"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066946    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6be1f4-198d-4b49-915a-364d0f852567" containerName="kserve-container"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066952    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066957    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066963    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066968    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066974    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="storage-initializer"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066979    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="storage-initializer"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066987    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="storage-initializer"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066993    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="storage-initializer"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.066999    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kube-rbac-proxy"
Jul 02 12:59:01.067001 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.067005    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kube-rbac-proxy"
Jul 02 12:59:01.067642 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.067068    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed6be1f4-198d-4b49-915a-364d0f852567" containerName="kserve-container"
Jul 02 12:59:01.067642 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.067075    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kserve-container"
Jul 02 12:59:01.067642 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.067083    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="kserve-container"
Jul 02 12:59:01.067642 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.067089    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df5f6ed-7cde-4590-b7d2-bd52620d608d" containerName="kube-rbac-proxy"
Jul 02 12:59:01.067642 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.067095    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed6be1f4-198d-4b49-915a-364d0f852567" containerName="kube-rbac-proxy"
Jul 02 12:59:01.067642 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.067100    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="kube-rbac-proxy"
Jul 02 12:59:01.067642 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.067104    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c52909b-7659-4633-bc89-3bbf0919ed3f" containerName="agent"
Jul 02 12:59:01.098332 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.098286    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"]
Jul 02 12:59:01.098476 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.098413    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.102945 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.102922    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 12:59:01.107190 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.104338    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 12:59:01.107190 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.104425    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 02 12:59:01.107190 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.104990    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 02 12:59:01.107190 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.105085    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rhzbx\""
Jul 02 12:59:01.107190 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.105652    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 12:59:01.107190 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.105853    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 12:59:01.181512 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.181479    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e631120f-884d-4d2e-b973-bd3fbbad14a6-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.181512 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.181513    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e631120f-884d-4d2e-b973-bd3fbbad14a6-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.181710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.181542    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmcvq\" (UniqueName: \"kubernetes.io/projected/e631120f-884d-4d2e-b973-bd3fbbad14a6-kube-api-access-tmcvq\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.181710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.181594    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e631120f-884d-4d2e-b973-bd3fbbad14a6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.181710 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.181669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e631120f-884d-4d2e-b973-bd3fbbad14a6-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.282167 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.282135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e631120f-884d-4d2e-b973-bd3fbbad14a6-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.282392 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.282180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e631120f-884d-4d2e-b973-bd3fbbad14a6-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.282392 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.282201    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e631120f-884d-4d2e-b973-bd3fbbad14a6-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.282392 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.282230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmcvq\" (UniqueName: \"kubernetes.io/projected/e631120f-884d-4d2e-b973-bd3fbbad14a6-kube-api-access-tmcvq\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.282392 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.282273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e631120f-884d-4d2e-b973-bd3fbbad14a6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.282628 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.282606    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e631120f-884d-4d2e-b973-bd3fbbad14a6-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.282973 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.282944    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e631120f-884d-4d2e-b973-bd3fbbad14a6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.282973 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.282958    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e631120f-884d-4d2e-b973-bd3fbbad14a6-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.284816 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.284794    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e631120f-884d-4d2e-b973-bd3fbbad14a6-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.291659 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.291635    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmcvq\" (UniqueName: \"kubernetes.io/projected/e631120f-884d-4d2e-b973-bd3fbbad14a6-kube-api-access-tmcvq\") pod \"isvc-autogluon-v2-predictor-67b849d456-bv7cm\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.411321 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.411222    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:01.573341 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.573315    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"]
Jul 02 12:59:01.575702 ip-10-0-143-149 kubenswrapper[2564]: W0702 12:59:01.575662    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode631120f_884d_4d2e_b973_bd3fbbad14a6.slice/crio-512c767ea138edf65e2a3323dcd122431ce360bee23cc2c80de98656c88aa2c5 WatchSource:0}: Error finding container 512c767ea138edf65e2a3323dcd122431ce360bee23cc2c80de98656c88aa2c5: Status 404 returned error can't find the container with id 512c767ea138edf65e2a3323dcd122431ce360bee23cc2c80de98656c88aa2c5
Jul 02 12:59:01.872134 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.872095    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm" event={"ID":"e631120f-884d-4d2e-b973-bd3fbbad14a6","Type":"ContainerStarted","Data":"633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3"}
Jul 02 12:59:01.872333 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:01.872155    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm" event={"ID":"e631120f-884d-4d2e-b973-bd3fbbad14a6","Type":"ContainerStarted","Data":"512c767ea138edf65e2a3323dcd122431ce360bee23cc2c80de98656c88aa2c5"}
Jul 02 12:59:32.004191 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:32.004154    2564 generic.go:356] "Generic (PLEG): container finished" podID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerID="633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3" exitCode=0
Jul 02 12:59:32.004191 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:32.004192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm" event={"ID":"e631120f-884d-4d2e-b973-bd3fbbad14a6","Type":"ContainerDied","Data":"633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3"}
Jul 02 12:59:33.011366 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:33.011325    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm" event={"ID":"e631120f-884d-4d2e-b973-bd3fbbad14a6","Type":"ContainerStarted","Data":"df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13"}
Jul 02 12:59:33.011862 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:33.011430    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm" event={"ID":"e631120f-884d-4d2e-b973-bd3fbbad14a6","Type":"ContainerStarted","Data":"b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1"}
Jul 02 12:59:33.011862 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:33.011647    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:34.016841 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:34.016810    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:40.027350 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:40.027323    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 12:59:40.052221 ip-10-0-143-149 kubenswrapper[2564]: I0702 12:59:40.052173    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm" podStartSLOduration=9.573324096 podStartE2EDuration="39.052159866s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.47883577s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:59:33.034158991 +0000 UTC m=+1221.017632675" watchObservedRunningTime="2026-07-02 12:59:40.052159866 +0000 UTC m=+1228.035633549"
Jul 02 13:01:08.422624 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:08.422595    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 13:01:11.541659 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:11.541625    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"]
Jul 02 13:01:11.542123 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:11.541894    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm" podUID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerName="kube-rbac-proxy" containerID="cri-o://df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13" gracePeriod=30
Jul 02 13:01:11.542123 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:11.541998    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm" podUID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerName="kserve-container" containerID="cri-o://b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1" gracePeriod=30
Jul 02 13:01:12.384957 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.384930    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 13:01:12.452598 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.452572    2564 generic.go:356] "Generic (PLEG): container finished" podID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerID="df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13" exitCode=2
Jul 02 13:01:12.452598 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.452594    2564 generic.go:356] "Generic (PLEG): container finished" podID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerID="b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1" exitCode=0
Jul 02 13:01:12.452787 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.452633    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"
Jul 02 13:01:12.452787 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.452635    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm" event={"ID":"e631120f-884d-4d2e-b973-bd3fbbad14a6","Type":"ContainerDied","Data":"df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13"}
Jul 02 13:01:12.452787 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.452686    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm" event={"ID":"e631120f-884d-4d2e-b973-bd3fbbad14a6","Type":"ContainerDied","Data":"b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1"}
Jul 02 13:01:12.452787 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.452705    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm" event={"ID":"e631120f-884d-4d2e-b973-bd3fbbad14a6","Type":"ContainerDied","Data":"512c767ea138edf65e2a3323dcd122431ce360bee23cc2c80de98656c88aa2c5"}
Jul 02 13:01:12.452787 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.452730    2564 scope.go:117] "RemoveContainer" containerID="df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13"
Jul 02 13:01:12.461623 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.461538    2564 scope.go:117] "RemoveContainer" containerID="b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1"
Jul 02 13:01:12.469104 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.469087    2564 scope.go:117] "RemoveContainer" containerID="633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3"
Jul 02 13:01:12.475986 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.475970    2564 scope.go:117] "RemoveContainer" containerID="df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13"
Jul 02 13:01:12.476228 ip-10-0-143-149 kubenswrapper[2564]: E0702 13:01:12.476209    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13\": container with ID starting with df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13 not found: ID does not exist" containerID="df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13"
Jul 02 13:01:12.476274 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.476237    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13"} err="failed to get container status \"df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13\": rpc error: code = NotFound desc = could not find container \"df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13\": container with ID starting with df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13 not found: ID does not exist"
Jul 02 13:01:12.476274 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.476259    2564 scope.go:117] "RemoveContainer" containerID="b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1"
Jul 02 13:01:12.476511 ip-10-0-143-149 kubenswrapper[2564]: E0702 13:01:12.476495    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1\": container with ID starting with b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1 not found: ID does not exist" containerID="b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1"
Jul 02 13:01:12.476562 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.476519    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1"} err="failed to get container status \"b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1\": rpc error: code = NotFound desc = could not find container \"b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1\": container with ID starting with b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1 not found: ID does not exist"
Jul 02 13:01:12.476562 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.476535    2564 scope.go:117] "RemoveContainer" containerID="633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3"
Jul 02 13:01:12.476748 ip-10-0-143-149 kubenswrapper[2564]: E0702 13:01:12.476731    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3\": container with ID starting with 633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3 not found: ID does not exist" containerID="633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3"
Jul 02 13:01:12.476790 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.476754    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3"} err="failed to get container status \"633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3\": rpc error: code = NotFound desc = could not find container \"633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3\": container with ID starting with 633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3 not found: ID does not exist"
Jul 02 13:01:12.476790 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.476770    2564 scope.go:117] "RemoveContainer" containerID="df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13"
Jul 02 13:01:12.476956 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.476941    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13"} err="failed to get container status \"df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13\": rpc error: code = NotFound desc = could not find container \"df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13\": container with ID starting with df0d864d6916e15c89d8e9aff3b00b3a57424e6a9b9ab778320f1d9409ecfc13 not found: ID does not exist"
Jul 02 13:01:12.477007 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.476957    2564 scope.go:117] "RemoveContainer" containerID="b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1"
Jul 02 13:01:12.477144 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.477129    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1"} err="failed to get container status \"b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1\": rpc error: code = NotFound desc = could not find container \"b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1\": container with ID starting with b1770d537baf3ea8cb5d72395aa4fa7aafeeee90143977484734e4f5430946a1 not found: ID does not exist"
Jul 02 13:01:12.477187 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.477145    2564 scope.go:117] "RemoveContainer" containerID="633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3"
Jul 02 13:01:12.477365 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.477339    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3"} err="failed to get container status \"633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3\": rpc error: code = NotFound desc = could not find container \"633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3\": container with ID starting with 633d3c4a2425a163994a1dfac0c7ebd75e5295982bd37d04d1cd50901b12bea3 not found: ID does not exist"
Jul 02 13:01:12.485655 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.485637    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmcvq\" (UniqueName: \"kubernetes.io/projected/e631120f-884d-4d2e-b973-bd3fbbad14a6-kube-api-access-tmcvq\") pod \"e631120f-884d-4d2e-b973-bd3fbbad14a6\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") "
Jul 02 13:01:12.485734 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.485670    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e631120f-884d-4d2e-b973-bd3fbbad14a6-proxy-tls\") pod \"e631120f-884d-4d2e-b973-bd3fbbad14a6\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") "
Jul 02 13:01:12.485880 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.485863    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e631120f-884d-4d2e-b973-bd3fbbad14a6-kserve-provision-location\") pod \"e631120f-884d-4d2e-b973-bd3fbbad14a6\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") "
Jul 02 13:01:12.485946 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.485931    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e631120f-884d-4d2e-b973-bd3fbbad14a6-kube-rbac-proxy-sar-config\") pod \"e631120f-884d-4d2e-b973-bd3fbbad14a6\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") "
Jul 02 13:01:12.485988 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.485972    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e631120f-884d-4d2e-b973-bd3fbbad14a6-cabundle-cert\") pod \"e631120f-884d-4d2e-b973-bd3fbbad14a6\" (UID: \"e631120f-884d-4d2e-b973-bd3fbbad14a6\") "
Jul 02 13:01:12.486342 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.486316    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e631120f-884d-4d2e-b973-bd3fbbad14a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e631120f-884d-4d2e-b973-bd3fbbad14a6" (UID: "e631120f-884d-4d2e-b973-bd3fbbad14a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:01:12.486440 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.486385    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e631120f-884d-4d2e-b973-bd3fbbad14a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e631120f-884d-4d2e-b973-bd3fbbad14a6" (UID: "e631120f-884d-4d2e-b973-bd3fbbad14a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:01:12.487650 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.487631    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e631120f-884d-4d2e-b973-bd3fbbad14a6-kube-api-access-tmcvq" (OuterVolumeSpecName: "kube-api-access-tmcvq") pod "e631120f-884d-4d2e-b973-bd3fbbad14a6" (UID: "e631120f-884d-4d2e-b973-bd3fbbad14a6"). InnerVolumeSpecName "kube-api-access-tmcvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:01:12.487911 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.487889    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e631120f-884d-4d2e-b973-bd3fbbad14a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e631120f-884d-4d2e-b973-bd3fbbad14a6" (UID: "e631120f-884d-4d2e-b973-bd3fbbad14a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:01:12.491021 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.490997    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e631120f-884d-4d2e-b973-bd3fbbad14a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e631120f-884d-4d2e-b973-bd3fbbad14a6" (UID: "e631120f-884d-4d2e-b973-bd3fbbad14a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:01:12.587051 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.587017    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmcvq\" (UniqueName: \"kubernetes.io/projected/e631120f-884d-4d2e-b973-bd3fbbad14a6-kube-api-access-tmcvq\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:01:12.587051 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.587049    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e631120f-884d-4d2e-b973-bd3fbbad14a6-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:01:12.587558 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.587063    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e631120f-884d-4d2e-b973-bd3fbbad14a6-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:01:12.587558 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.587076    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e631120f-884d-4d2e-b973-bd3fbbad14a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:01:12.587558 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.587090    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e631120f-884d-4d2e-b973-bd3fbbad14a6-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:01:12.780787 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.780757    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"]
Jul 02 13:01:12.786410 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:12.786387    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-bv7cm"]
Jul 02 13:01:14.423311 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:14.423258    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e631120f-884d-4d2e-b973-bd3fbbad14a6" path="/var/lib/kubelet/pods/e631120f-884d-4d2e-b973-bd3fbbad14a6/volumes"
Jul 02 13:01:16.439451 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.439405    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"]
Jul 02 13:01:16.439852 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.439730    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerName="kube-rbac-proxy"
Jul 02 13:01:16.439852 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.439741    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerName="kube-rbac-proxy"
Jul 02 13:01:16.439852 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.439752    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerName="storage-initializer"
Jul 02 13:01:16.439852 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.439757    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerName="storage-initializer"
Jul 02 13:01:16.439852 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.439765    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerName="kserve-container"
Jul 02 13:01:16.439852 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.439772    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerName="kserve-container"
Jul 02 13:01:16.439852 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.439834    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerName="kserve-container"
Jul 02 13:01:16.439852 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.439840    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e631120f-884d-4d2e-b973-bd3fbbad14a6" containerName="kube-rbac-proxy"
Jul 02 13:01:16.441978 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.441958    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.448728 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.448702    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 02 13:01:16.449150 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.449133    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:01:16.449218 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.449190    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:01:16.449942 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.449911    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:01:16.450066 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.449952    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 02 13:01:16.450066 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.449963    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:01:16.450499 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.449947    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rhzbx\""
Jul 02 13:01:16.468405 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.468383    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"]
Jul 02 13:01:16.622626 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.622590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.622805 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.622635    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqt9c\" (UniqueName: \"kubernetes.io/projected/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kube-api-access-kqt9c\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.622805 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.622713    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.622897 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.622806    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.622897 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.622836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.724190 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.724097    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.724190 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.724146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqt9c\" (UniqueName: \"kubernetes.io/projected/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kube-api-access-kqt9c\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.724448 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.724201    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.724448 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.724260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.724448 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.724286    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.724660 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.724640    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.724960 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.724937    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.725042 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.724967    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.726807 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.726784    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.745016 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.744993    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqt9c\" (UniqueName: \"kubernetes.io/projected/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kube-api-access-kqt9c\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.754388 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.754359    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:16.883665 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:16.883644    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"]
Jul 02 13:01:16.886071 ip-10-0-143-149 kubenswrapper[2564]: W0702 13:01:16.886045    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936a02a2_f7f7_46d9_bf3f_70c9917e4e82.slice/crio-4057a95c0f21b9cea0d328b736d3be6fc46c6f558ea910f0e0f64d1a165c96f5 WatchSource:0}: Error finding container 4057a95c0f21b9cea0d328b736d3be6fc46c6f558ea910f0e0f64d1a165c96f5: Status 404 returned error can't find the container with id 4057a95c0f21b9cea0d328b736d3be6fc46c6f558ea910f0e0f64d1a165c96f5
Jul 02 13:01:17.476231 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:17.476195    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p" event={"ID":"936a02a2-f7f7-46d9-bf3f-70c9917e4e82","Type":"ContainerStarted","Data":"24193198154dcb3cfbdbb8998ddc7df186ec0780d2c7d38e428e73ef9160872d"}
Jul 02 13:01:17.476652 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:17.476251    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p" event={"ID":"936a02a2-f7f7-46d9-bf3f-70c9917e4e82","Type":"ContainerStarted","Data":"4057a95c0f21b9cea0d328b736d3be6fc46c6f558ea910f0e0f64d1a165c96f5"}
Jul 02 13:01:22.498357 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:22.498321    2564 generic.go:356] "Generic (PLEG): container finished" podID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerID="24193198154dcb3cfbdbb8998ddc7df186ec0780d2c7d38e428e73ef9160872d" exitCode=0
Jul 02 13:01:22.498704 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:22.498397    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p" event={"ID":"936a02a2-f7f7-46d9-bf3f-70c9917e4e82","Type":"ContainerDied","Data":"24193198154dcb3cfbdbb8998ddc7df186ec0780d2c7d38e428e73ef9160872d"}
Jul 02 13:01:23.505073 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:23.505031    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p" event={"ID":"936a02a2-f7f7-46d9-bf3f-70c9917e4e82","Type":"ContainerStarted","Data":"9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f"}
Jul 02 13:01:23.505487 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:23.505116    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p" event={"ID":"936a02a2-f7f7-46d9-bf3f-70c9917e4e82","Type":"ContainerStarted","Data":"28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be"}
Jul 02 13:01:23.505487 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:23.505355    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:23.530918 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:23.530865    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p" podStartSLOduration=2.771066189 podStartE2EDuration="7.530850404s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.759784215s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:01:23.529640766 +0000 UTC m=+1331.513114449" watchObservedRunningTime="2026-07-02 13:01:23.530850404 +0000 UTC m=+1331.514324086"
Jul 02 13:01:24.509387 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:24.509356    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:01:30.520066 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:01:30.520033    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:03:00.423065 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:00.423039    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:03:06.543653 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:06.543559    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"]
Jul 02 13:03:06.544162 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:06.543836    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p" podUID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerName="kserve-container" containerID="cri-o://28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be" gracePeriod=30
Jul 02 13:03:06.544162 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:06.543922    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p" podUID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerName="kube-rbac-proxy" containerID="cri-o://9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f" gracePeriod=30
Jul 02 13:03:06.965043 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:06.965009    2564 generic.go:356] "Generic (PLEG): container finished" podID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerID="9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f" exitCode=2
Jul 02 13:03:06.965221 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:06.965088    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p" event={"ID":"936a02a2-f7f7-46d9-bf3f-70c9917e4e82","Type":"ContainerDied","Data":"9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f"}
Jul 02 13:03:07.297049 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.297027    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:03:07.444705 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.444680    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kube-rbac-proxy-sar-config\") pod \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") "
Jul 02 13:03:07.444833 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.444715    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-cabundle-cert\") pod \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") "
Jul 02 13:03:07.444833 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.444765    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kserve-provision-location\") pod \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") "
Jul 02 13:03:07.444833 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.444797    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-proxy-tls\") pod \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") "
Jul 02 13:03:07.444997 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.444833    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqt9c\" (UniqueName: \"kubernetes.io/projected/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kube-api-access-kqt9c\") pod \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\" (UID: \"936a02a2-f7f7-46d9-bf3f-70c9917e4e82\") "
Jul 02 13:03:07.445110 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.445076    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "936a02a2-f7f7-46d9-bf3f-70c9917e4e82" (UID: "936a02a2-f7f7-46d9-bf3f-70c9917e4e82"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:03:07.445218 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.445132    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "936a02a2-f7f7-46d9-bf3f-70c9917e4e82" (UID: "936a02a2-f7f7-46d9-bf3f-70c9917e4e82"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:03:07.446875 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.446845    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kube-api-access-kqt9c" (OuterVolumeSpecName: "kube-api-access-kqt9c") pod "936a02a2-f7f7-46d9-bf3f-70c9917e4e82" (UID: "936a02a2-f7f7-46d9-bf3f-70c9917e4e82"). InnerVolumeSpecName "kube-api-access-kqt9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:03:07.446951 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.446936    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "936a02a2-f7f7-46d9-bf3f-70c9917e4e82" (UID: "936a02a2-f7f7-46d9-bf3f-70c9917e4e82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:03:07.450600 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.450577    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "936a02a2-f7f7-46d9-bf3f-70c9917e4e82" (UID: "936a02a2-f7f7-46d9-bf3f-70c9917e4e82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:03:07.545590 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.545502    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:03:07.545590 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.545540    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:03:07.545590 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.545549    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:03:07.545590 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.545561    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:03:07.545590 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.545569    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqt9c\" (UniqueName: \"kubernetes.io/projected/936a02a2-f7f7-46d9-bf3f-70c9917e4e82-kube-api-access-kqt9c\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:03:07.970849 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.970817    2564 generic.go:356] "Generic (PLEG): container finished" podID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerID="28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be" exitCode=0
Jul 02 13:03:07.971007 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.970871    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"
Jul 02 13:03:07.971007 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.970904    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p" event={"ID":"936a02a2-f7f7-46d9-bf3f-70c9917e4e82","Type":"ContainerDied","Data":"28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be"}
Jul 02 13:03:07.971007 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.970958    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p" event={"ID":"936a02a2-f7f7-46d9-bf3f-70c9917e4e82","Type":"ContainerDied","Data":"4057a95c0f21b9cea0d328b736d3be6fc46c6f558ea910f0e0f64d1a165c96f5"}
Jul 02 13:03:07.971007 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.970984    2564 scope.go:117] "RemoveContainer" containerID="9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f"
Jul 02 13:03:07.982641 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.982621    2564 scope.go:117] "RemoveContainer" containerID="28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be"
Jul 02 13:03:07.990063 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.990045    2564 scope.go:117] "RemoveContainer" containerID="24193198154dcb3cfbdbb8998ddc7df186ec0780d2c7d38e428e73ef9160872d"
Jul 02 13:03:07.997114 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.997081    2564 scope.go:117] "RemoveContainer" containerID="9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f"
Jul 02 13:03:07.997661 ip-10-0-143-149 kubenswrapper[2564]: E0702 13:03:07.997404    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f\": container with ID starting with 9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f not found: ID does not exist" containerID="9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f"
Jul 02 13:03:07.997661 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.997437    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f"} err="failed to get container status \"9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f\": rpc error: code = NotFound desc = could not find container \"9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f\": container with ID starting with 9279e9c2465467c18d658e9b0d704187c87d63958a0408bec6a9d80ebd1a115f not found: ID does not exist"
Jul 02 13:03:07.997661 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.997460    2564 scope.go:117] "RemoveContainer" containerID="28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be"
Jul 02 13:03:07.997864 ip-10-0-143-149 kubenswrapper[2564]: E0702 13:03:07.997846    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be\": container with ID starting with 28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be not found: ID does not exist" containerID="28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be"
Jul 02 13:03:07.997912 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.997872    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be"} err="failed to get container status \"28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be\": rpc error: code = NotFound desc = could not find container \"28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be\": container with ID starting with 28267f6b87bef6b41f0cc61d39dc3747be55f1b240dc24ff6515d196354638be not found: ID does not exist"
Jul 02 13:03:07.997912 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.997887    2564 scope.go:117] "RemoveContainer" containerID="24193198154dcb3cfbdbb8998ddc7df186ec0780d2c7d38e428e73ef9160872d"
Jul 02 13:03:07.998163 ip-10-0-143-149 kubenswrapper[2564]: E0702 13:03:07.998138    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24193198154dcb3cfbdbb8998ddc7df186ec0780d2c7d38e428e73ef9160872d\": container with ID starting with 24193198154dcb3cfbdbb8998ddc7df186ec0780d2c7d38e428e73ef9160872d not found: ID does not exist" containerID="24193198154dcb3cfbdbb8998ddc7df186ec0780d2c7d38e428e73ef9160872d"
Jul 02 13:03:07.998218 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:07.998171    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24193198154dcb3cfbdbb8998ddc7df186ec0780d2c7d38e428e73ef9160872d"} err="failed to get container status \"24193198154dcb3cfbdbb8998ddc7df186ec0780d2c7d38e428e73ef9160872d\": rpc error: code = NotFound desc = could not find container \"24193198154dcb3cfbdbb8998ddc7df186ec0780d2c7d38e428e73ef9160872d\": container with ID starting with 24193198154dcb3cfbdbb8998ddc7df186ec0780d2c7d38e428e73ef9160872d not found: ID does not exist"
Jul 02 13:03:08.000766 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:08.000741    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"]
Jul 02 13:03:08.006089 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:08.006070    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-dx54p"]
Jul 02 13:03:08.423541 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:08.423458    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" path="/var/lib/kubelet/pods/936a02a2-f7f7-46d9-bf3f-70c9917e4e82/volumes"
Jul 02 13:03:11.694897 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.694868    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"]
Jul 02 13:03:11.695280 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.695210    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerName="storage-initializer"
Jul 02 13:03:11.695280 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.695222    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerName="storage-initializer"
Jul 02 13:03:11.695280 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.695238    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerName="kserve-container"
Jul 02 13:03:11.695280 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.695243    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerName="kserve-container"
Jul 02 13:03:11.695280 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.695255    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerName="kube-rbac-proxy"
Jul 02 13:03:11.695280 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.695261    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerName="kube-rbac-proxy"
Jul 02 13:03:11.695509 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.695330    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerName="kube-rbac-proxy"
Jul 02 13:03:11.695509 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.695342    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="936a02a2-f7f7-46d9-bf3f-70c9917e4e82" containerName="kserve-container"
Jul 02 13:03:11.700356 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.700336    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.703483 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.703461    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:03:11.703591 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.703482    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 02 13:03:11.704688 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.704669    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:03:11.704805 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.704690    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rhzbx\""
Jul 02 13:03:11.704805 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.704729    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:03:11.704805 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.704668    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:03:11.704805 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.704668    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 02 13:03:11.711408 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.711388    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"]
Jul 02 13:03:11.883753 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.883714    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.883899 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.883772    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.883899 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.883806    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.883899 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.883848    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbthf\" (UniqueName: \"kubernetes.io/projected/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kube-api-access-mbthf\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.884007 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.883920    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.984708 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.984631    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.984708 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.984671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.984708 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.984708    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbthf\" (UniqueName: \"kubernetes.io/projected/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kube-api-access-mbthf\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.984916 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.984739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.984916 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.984764    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.985233 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.985215    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.985489 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.985462    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.985489 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.985480    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.987269 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.987224    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:11.995064 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:11.995033    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbthf\" (UniqueName: \"kubernetes.io/projected/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kube-api-access-mbthf\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:12.011727 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:12.011705    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:12.145037 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:12.145014    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"]
Jul 02 13:03:12.147237 ip-10-0-143-149 kubenswrapper[2564]: W0702 13:03:12.147207    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65f6542_4c00_4c7c_b982_2d64d15a8cbe.slice/crio-69586921fd2975a05e8871c01b49d45eeea99ea579deee4cc927180d061e5552 WatchSource:0}: Error finding container 69586921fd2975a05e8871c01b49d45eeea99ea579deee4cc927180d061e5552: Status 404 returned error can't find the container with id 69586921fd2975a05e8871c01b49d45eeea99ea579deee4cc927180d061e5552
Jul 02 13:03:12.996902 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:12.996862    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf" event={"ID":"a65f6542-4c00-4c7c-b982-2d64d15a8cbe","Type":"ContainerStarted","Data":"092cc66b3a7109444dbbd945b0ec427f2b997ccd9da1b4f61e9efaa2d24e5d76"}
Jul 02 13:03:12.997223 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:12.996939    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf" event={"ID":"a65f6542-4c00-4c7c-b982-2d64d15a8cbe","Type":"ContainerStarted","Data":"69586921fd2975a05e8871c01b49d45eeea99ea579deee4cc927180d061e5552"}
Jul 02 13:03:42.120688 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:42.120656    2564 generic.go:356] "Generic (PLEG): container finished" podID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerID="092cc66b3a7109444dbbd945b0ec427f2b997ccd9da1b4f61e9efaa2d24e5d76" exitCode=0
Jul 02 13:03:42.121100 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:42.120729    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf" event={"ID":"a65f6542-4c00-4c7c-b982-2d64d15a8cbe","Type":"ContainerDied","Data":"092cc66b3a7109444dbbd945b0ec427f2b997ccd9da1b4f61e9efaa2d24e5d76"}
Jul 02 13:03:43.127202 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:43.127165    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf" event={"ID":"a65f6542-4c00-4c7c-b982-2d64d15a8cbe","Type":"ContainerStarted","Data":"59db94bb42d4c2c7941e8d6629bff11e96bc3d8f2c36efe597e1ee4558ac2019"}
Jul 02 13:03:43.127622 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:43.127228    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf" event={"ID":"a65f6542-4c00-4c7c-b982-2d64d15a8cbe","Type":"ContainerStarted","Data":"ae5b55b4237e5846724c87f2648a54bea9b5857e665c5ec725eeaca839c16728"}
Jul 02 13:03:43.127622 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:43.127471    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:43.127710 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:43.127633    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:03:43.152340 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:43.152264    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf" podStartSLOduration=2.9953759250000003 podStartE2EDuration="32.152245026s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.156869101s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:03:43.149248019 +0000 UTC m=+1471.132721704" watchObservedRunningTime="2026-07-02 13:03:43.152245026 +0000 UTC m=+1471.135718710"
Jul 02 13:03:49.136269 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:03:49.136241    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:05:16.422871 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:16.422839    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:05:21.951494 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:21.951463    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"]
Jul 02 13:05:21.951881 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:21.951701    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf" podUID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerName="kserve-container" containerID="cri-o://ae5b55b4237e5846724c87f2648a54bea9b5857e665c5ec725eeaca839c16728" gracePeriod=30
Jul 02 13:05:21.951881 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:21.951735    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf" podUID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerName="kube-rbac-proxy" containerID="cri-o://59db94bb42d4c2c7941e8d6629bff11e96bc3d8f2c36efe597e1ee4558ac2019" gracePeriod=30
Jul 02 13:05:22.574717 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.574682    2564 generic.go:356] "Generic (PLEG): container finished" podID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerID="59db94bb42d4c2c7941e8d6629bff11e96bc3d8f2c36efe597e1ee4558ac2019" exitCode=2
Jul 02 13:05:22.574717 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.574705    2564 generic.go:356] "Generic (PLEG): container finished" podID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerID="ae5b55b4237e5846724c87f2648a54bea9b5857e665c5ec725eeaca839c16728" exitCode=0
Jul 02 13:05:22.574870 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.574752    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf" event={"ID":"a65f6542-4c00-4c7c-b982-2d64d15a8cbe","Type":"ContainerDied","Data":"59db94bb42d4c2c7941e8d6629bff11e96bc3d8f2c36efe597e1ee4558ac2019"}
Jul 02 13:05:22.574870 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.574811    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf" event={"ID":"a65f6542-4c00-4c7c-b982-2d64d15a8cbe","Type":"ContainerDied","Data":"ae5b55b4237e5846724c87f2648a54bea9b5857e665c5ec725eeaca839c16728"}
Jul 02 13:05:22.697082 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.697061    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:05:22.764828 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.764805    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-cabundle-cert\") pod \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") "
Jul 02 13:05:22.764960 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.764840    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-proxy-tls\") pod \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") "
Jul 02 13:05:22.765041 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.765023    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbthf\" (UniqueName: \"kubernetes.io/projected/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kube-api-access-mbthf\") pod \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") "
Jul 02 13:05:22.765100 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.765084    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kube-rbac-proxy-sar-config\") pod \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") "
Jul 02 13:05:22.765165 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.765120    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kserve-provision-location\") pod \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\" (UID: \"a65f6542-4c00-4c7c-b982-2d64d15a8cbe\") "
Jul 02 13:05:22.765226 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.765195    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a65f6542-4c00-4c7c-b982-2d64d15a8cbe" (UID: "a65f6542-4c00-4c7c-b982-2d64d15a8cbe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:05:22.765453 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.765430    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:05:22.765561 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.765523    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a65f6542-4c00-4c7c-b982-2d64d15a8cbe" (UID: "a65f6542-4c00-4c7c-b982-2d64d15a8cbe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:05:22.766890 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.766867    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a65f6542-4c00-4c7c-b982-2d64d15a8cbe" (UID: "a65f6542-4c00-4c7c-b982-2d64d15a8cbe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:05:22.767063 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.767049    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kube-api-access-mbthf" (OuterVolumeSpecName: "kube-api-access-mbthf") pod "a65f6542-4c00-4c7c-b982-2d64d15a8cbe" (UID: "a65f6542-4c00-4c7c-b982-2d64d15a8cbe"). InnerVolumeSpecName "kube-api-access-mbthf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:05:22.769751 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.769729    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a65f6542-4c00-4c7c-b982-2d64d15a8cbe" (UID: "a65f6542-4c00-4c7c-b982-2d64d15a8cbe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:05:22.866731 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.866674    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:05:22.866731 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.866696    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:05:22.866731 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.866706    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:05:22.866731 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:22.866715    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbthf\" (UniqueName: \"kubernetes.io/projected/a65f6542-4c00-4c7c-b982-2d64d15a8cbe-kube-api-access-mbthf\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:05:23.580521 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:23.580494    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"
Jul 02 13:05:23.580957 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:23.580493    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf" event={"ID":"a65f6542-4c00-4c7c-b982-2d64d15a8cbe","Type":"ContainerDied","Data":"69586921fd2975a05e8871c01b49d45eeea99ea579deee4cc927180d061e5552"}
Jul 02 13:05:23.580957 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:23.580633    2564 scope.go:117] "RemoveContainer" containerID="59db94bb42d4c2c7941e8d6629bff11e96bc3d8f2c36efe597e1ee4558ac2019"
Jul 02 13:05:23.589430 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:23.589413    2564 scope.go:117] "RemoveContainer" containerID="ae5b55b4237e5846724c87f2648a54bea9b5857e665c5ec725eeaca839c16728"
Jul 02 13:05:23.596506 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:23.596490    2564 scope.go:117] "RemoveContainer" containerID="092cc66b3a7109444dbbd945b0ec427f2b997ccd9da1b4f61e9efaa2d24e5d76"
Jul 02 13:05:23.604362 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:23.604343    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"]
Jul 02 13:05:23.609076 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:23.609056    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4vnbf"]
Jul 02 13:05:24.422901 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:24.422863    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" path="/var/lib/kubelet/pods/a65f6542-4c00-4c7c-b982-2d64d15a8cbe/volumes"
Jul 02 13:05:27.057488 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.057454    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"]
Jul 02 13:05:27.057930 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.057916    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerName="kube-rbac-proxy"
Jul 02 13:05:27.057974 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.057933    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerName="kube-rbac-proxy"
Jul 02 13:05:27.057974 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.057948    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerName="storage-initializer"
Jul 02 13:05:27.057974 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.057957    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerName="storage-initializer"
Jul 02 13:05:27.057974 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.057968    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerName="kserve-container"
Jul 02 13:05:27.058094 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.057977    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerName="kserve-container"
Jul 02 13:05:27.058094 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.058083    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerName="kserve-container"
Jul 02 13:05:27.058157 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.058100    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65f6542-4c00-4c7c-b982-2d64d15a8cbe" containerName="kube-rbac-proxy"
Jul 02 13:05:27.063484 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.063466    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.066213 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.066051    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 02 13:05:27.066213 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.066076    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:05:27.066213 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.066096    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:05:27.066213 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.066134    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:05:27.066213 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.066166    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:05:27.066213 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.066200    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rhzbx\""
Jul 02 13:05:27.066584 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.066323    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 02 13:05:27.070128 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.070107    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"]
Jul 02 13:05:27.101789 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.101761    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/243651c4-4b41-4814-8e89-c85de903be78-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.101789 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.101788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/243651c4-4b41-4814-8e89-c85de903be78-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.101980 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.101810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/243651c4-4b41-4814-8e89-c85de903be78-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.101980 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.101900    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/243651c4-4b41-4814-8e89-c85de903be78-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.101980 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.101938    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bfk\" (UniqueName: \"kubernetes.io/projected/243651c4-4b41-4814-8e89-c85de903be78-kube-api-access-p4bfk\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.202954 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.202928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/243651c4-4b41-4814-8e89-c85de903be78-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.203107 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.202959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bfk\" (UniqueName: \"kubernetes.io/projected/243651c4-4b41-4814-8e89-c85de903be78-kube-api-access-p4bfk\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.203107 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.203003    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/243651c4-4b41-4814-8e89-c85de903be78-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.203107 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.203021    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/243651c4-4b41-4814-8e89-c85de903be78-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.203107 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.203039    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/243651c4-4b41-4814-8e89-c85de903be78-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.203463 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.203445    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/243651c4-4b41-4814-8e89-c85de903be78-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.203642 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.203625    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/243651c4-4b41-4814-8e89-c85de903be78-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.203707 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.203690    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/243651c4-4b41-4814-8e89-c85de903be78-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.205486 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.205470    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/243651c4-4b41-4814-8e89-c85de903be78-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.211553 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.211533    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bfk\" (UniqueName: \"kubernetes.io/projected/243651c4-4b41-4814-8e89-c85de903be78-kube-api-access-p4bfk\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.374669 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.374585    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:27.510023 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.509997    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"]
Jul 02 13:05:27.512529 ip-10-0-143-149 kubenswrapper[2564]: W0702 13:05:27.512503    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243651c4_4b41_4814_8e89_c85de903be78.slice/crio-285e2db1bc7199242fd8e99e913b9a167ffad8bb023b79c5c9c19fa4c1326a12 WatchSource:0}: Error finding container 285e2db1bc7199242fd8e99e913b9a167ffad8bb023b79c5c9c19fa4c1326a12: Status 404 returned error can't find the container with id 285e2db1bc7199242fd8e99e913b9a167ffad8bb023b79c5c9c19fa4c1326a12
Jul 02 13:05:27.598812 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.598779    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" event={"ID":"243651c4-4b41-4814-8e89-c85de903be78","Type":"ContainerStarted","Data":"ed19a363d83b90b4fb63027601574a77a8677600d297bcb9ce31c3b6ba5d34de"}
Jul 02 13:05:27.598950 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:27.598837    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" event={"ID":"243651c4-4b41-4814-8e89-c85de903be78","Type":"ContainerStarted","Data":"285e2db1bc7199242fd8e99e913b9a167ffad8bb023b79c5c9c19fa4c1326a12"}
Jul 02 13:05:37.643928 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:37.643838    2564 generic.go:356] "Generic (PLEG): container finished" podID="243651c4-4b41-4814-8e89-c85de903be78" containerID="ed19a363d83b90b4fb63027601574a77a8677600d297bcb9ce31c3b6ba5d34de" exitCode=0
Jul 02 13:05:37.643928 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:37.643898    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" event={"ID":"243651c4-4b41-4814-8e89-c85de903be78","Type":"ContainerDied","Data":"ed19a363d83b90b4fb63027601574a77a8677600d297bcb9ce31c3b6ba5d34de"}
Jul 02 13:05:38.651471 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:38.651441    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" event={"ID":"243651c4-4b41-4814-8e89-c85de903be78","Type":"ContainerStarted","Data":"6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6"}
Jul 02 13:05:38.651859 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:38.651500    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" event={"ID":"243651c4-4b41-4814-8e89-c85de903be78","Type":"ContainerStarted","Data":"a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac"}
Jul 02 13:05:38.651859 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:38.651686    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:38.672896 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:38.672853    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" podStartSLOduration=1.871577917 podStartE2EDuration="11.672840255s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.801262338s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:05:38.671929556 +0000 UTC m=+1586.655403250" watchObservedRunningTime="2026-07-02 13:05:38.672840255 +0000 UTC m=+1586.656313937"
Jul 02 13:05:39.656622 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:39.656587    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:39.658032 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:39.658004    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 13:05:40.662070 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:40.662038    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 13:05:45.666906 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:45.666879    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:45.667399 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:45.667370    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 13:05:55.668464 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:55.668432    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:57.146249 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:57.146215    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"]
Jul 02 13:05:57.146681 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:57.146459    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="kserve-container" containerID="cri-o://a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac" gracePeriod=30
Jul 02 13:05:57.146681 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:57.146544    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="kube-rbac-proxy" containerID="cri-o://6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6" gracePeriod=30
Jul 02 13:05:57.747526 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:57.747490    2564 generic.go:356] "Generic (PLEG): container finished" podID="243651c4-4b41-4814-8e89-c85de903be78" containerID="6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6" exitCode=2
Jul 02 13:05:57.747681 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:57.747564    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" event={"ID":"243651c4-4b41-4814-8e89-c85de903be78","Type":"ContainerDied","Data":"6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6"}
Jul 02 13:05:58.088105 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.088083    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:58.184505 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.184476    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4bfk\" (UniqueName: \"kubernetes.io/projected/243651c4-4b41-4814-8e89-c85de903be78-kube-api-access-p4bfk\") pod \"243651c4-4b41-4814-8e89-c85de903be78\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") "
Jul 02 13:05:58.184988 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.184537    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/243651c4-4b41-4814-8e89-c85de903be78-proxy-tls\") pod \"243651c4-4b41-4814-8e89-c85de903be78\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") "
Jul 02 13:05:58.184988 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.184557    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/243651c4-4b41-4814-8e89-c85de903be78-kube-rbac-proxy-sar-config\") pod \"243651c4-4b41-4814-8e89-c85de903be78\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") "
Jul 02 13:05:58.184988 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.184588    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/243651c4-4b41-4814-8e89-c85de903be78-kserve-provision-location\") pod \"243651c4-4b41-4814-8e89-c85de903be78\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") "
Jul 02 13:05:58.184988 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.184619    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/243651c4-4b41-4814-8e89-c85de903be78-cabundle-cert\") pod \"243651c4-4b41-4814-8e89-c85de903be78\" (UID: \"243651c4-4b41-4814-8e89-c85de903be78\") "
Jul 02 13:05:58.185161 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.184994    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243651c4-4b41-4814-8e89-c85de903be78-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "243651c4-4b41-4814-8e89-c85de903be78" (UID: "243651c4-4b41-4814-8e89-c85de903be78"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:05:58.185161 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.185064    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243651c4-4b41-4814-8e89-c85de903be78-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "243651c4-4b41-4814-8e89-c85de903be78" (UID: "243651c4-4b41-4814-8e89-c85de903be78"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:05:58.185650 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.185627    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243651c4-4b41-4814-8e89-c85de903be78-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "243651c4-4b41-4814-8e89-c85de903be78" (UID: "243651c4-4b41-4814-8e89-c85de903be78"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:05:58.186579 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.186557    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243651c4-4b41-4814-8e89-c85de903be78-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "243651c4-4b41-4814-8e89-c85de903be78" (UID: "243651c4-4b41-4814-8e89-c85de903be78"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:05:58.186723 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.186703    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243651c4-4b41-4814-8e89-c85de903be78-kube-api-access-p4bfk" (OuterVolumeSpecName: "kube-api-access-p4bfk") pod "243651c4-4b41-4814-8e89-c85de903be78" (UID: "243651c4-4b41-4814-8e89-c85de903be78"). InnerVolumeSpecName "kube-api-access-p4bfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:05:58.286108 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.286002    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bfk\" (UniqueName: \"kubernetes.io/projected/243651c4-4b41-4814-8e89-c85de903be78-kube-api-access-p4bfk\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:05:58.286108 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.286047    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/243651c4-4b41-4814-8e89-c85de903be78-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:05:58.286108 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.286061    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/243651c4-4b41-4814-8e89-c85de903be78-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:05:58.286108 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.286074    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/243651c4-4b41-4814-8e89-c85de903be78-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:05:58.286108 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.286086    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/243651c4-4b41-4814-8e89-c85de903be78-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Jul 02 13:05:58.754182 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.754149    2564 generic.go:356] "Generic (PLEG): container finished" podID="243651c4-4b41-4814-8e89-c85de903be78" containerID="a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac" exitCode=0
Jul 02 13:05:58.754378 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.754212    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" event={"ID":"243651c4-4b41-4814-8e89-c85de903be78","Type":"ContainerDied","Data":"a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac"}
Jul 02 13:05:58.754378 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.754214    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"
Jul 02 13:05:58.754378 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.754273    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc" event={"ID":"243651c4-4b41-4814-8e89-c85de903be78","Type":"ContainerDied","Data":"285e2db1bc7199242fd8e99e913b9a167ffad8bb023b79c5c9c19fa4c1326a12"}
Jul 02 13:05:58.754378 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.754335    2564 scope.go:117] "RemoveContainer" containerID="6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6"
Jul 02 13:05:58.762513 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.762492    2564 scope.go:117] "RemoveContainer" containerID="a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac"
Jul 02 13:05:58.769461 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.769444    2564 scope.go:117] "RemoveContainer" containerID="ed19a363d83b90b4fb63027601574a77a8677600d297bcb9ce31c3b6ba5d34de"
Jul 02 13:05:58.773818 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.773797    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"]
Jul 02 13:05:58.777056 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.777018    2564 scope.go:117] "RemoveContainer" containerID="6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6"
Jul 02 13:05:58.777367 ip-10-0-143-149 kubenswrapper[2564]: E0702 13:05:58.777338    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6\": container with ID starting with 6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6 not found: ID does not exist" containerID="6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6"
Jul 02 13:05:58.777459 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.777376    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6"} err="failed to get container status \"6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6\": rpc error: code = NotFound desc = could not find container \"6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6\": container with ID starting with 6aa83b19a13cd40e1c0891d4936ad402b8215730ca2ebbde8761459253382ed6 not found: ID does not exist"
Jul 02 13:05:58.777459 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.777398    2564 scope.go:117] "RemoveContainer" containerID="a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac"
Jul 02 13:05:58.777643 ip-10-0-143-149 kubenswrapper[2564]: E0702 13:05:58.777623    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac\": container with ID starting with a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac not found: ID does not exist" containerID="a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac"
Jul 02 13:05:58.777710 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.777653    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac"} err="failed to get container status \"a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac\": rpc error: code = NotFound desc = could not find container \"a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac\": container with ID starting with a29b69079948e82108c0fba86c6bd191331d73dcc747f7f81fbcb5a7692dc3ac not found: ID does not exist"
Jul 02 13:05:58.777710 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.777676    2564 scope.go:117] "RemoveContainer" containerID="ed19a363d83b90b4fb63027601574a77a8677600d297bcb9ce31c3b6ba5d34de"
Jul 02 13:05:58.777955 ip-10-0-143-149 kubenswrapper[2564]: E0702 13:05:58.777936    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed19a363d83b90b4fb63027601574a77a8677600d297bcb9ce31c3b6ba5d34de\": container with ID starting with ed19a363d83b90b4fb63027601574a77a8677600d297bcb9ce31c3b6ba5d34de not found: ID does not exist" containerID="ed19a363d83b90b4fb63027601574a77a8677600d297bcb9ce31c3b6ba5d34de"
Jul 02 13:05:58.778004 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.777961    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed19a363d83b90b4fb63027601574a77a8677600d297bcb9ce31c3b6ba5d34de"} err="failed to get container status \"ed19a363d83b90b4fb63027601574a77a8677600d297bcb9ce31c3b6ba5d34de\": rpc error: code = NotFound desc = could not find container \"ed19a363d83b90b4fb63027601574a77a8677600d297bcb9ce31c3b6ba5d34de\": container with ID starting with ed19a363d83b90b4fb63027601574a77a8677600d297bcb9ce31c3b6ba5d34de not found: ID does not exist"
Jul 02 13:05:58.778771 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:05:58.778753    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-r9rnc"]
Jul 02 13:06:00.422833 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:00.422800    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="243651c4-4b41-4814-8e89-c85de903be78" path="/var/lib/kubelet/pods/243651c4-4b41-4814-8e89-c85de903be78/volumes"
Jul 02 13:06:32.093142 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:32.093112    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bx9gl_1da27b61-6566-4583-a851-61ea1f1e8d11/global-pull-secret-syncer/0.log"
Jul 02 13:06:32.267860 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:32.267811    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nmzpg_dd02445c-74cd-43da-9315-6e6cf4e5050f/konnectivity-agent/0.log"
Jul 02 13:06:32.488147 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:32.488118    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-149.ec2.internal_6a9ee241c13dace2d3c74948d3ea9c6f/haproxy/0.log"
Jul 02 13:06:35.997617 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:35.997585    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6f21957-0de2-44f3-9625-5e208e2bb4be/alertmanager/0.log"
Jul 02 13:06:36.088101 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:36.088070    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6f21957-0de2-44f3-9625-5e208e2bb4be/config-reloader/0.log"
Jul 02 13:06:36.153210 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:36.153185    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6f21957-0de2-44f3-9625-5e208e2bb4be/kube-rbac-proxy-web/0.log"
Jul 02 13:06:36.229255 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:36.229231    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6f21957-0de2-44f3-9625-5e208e2bb4be/kube-rbac-proxy/0.log"
Jul 02 13:06:36.332755 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:36.332688    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6f21957-0de2-44f3-9625-5e208e2bb4be/kube-rbac-proxy-metric/0.log"
Jul 02 13:06:36.405625 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:36.405597    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6f21957-0de2-44f3-9625-5e208e2bb4be/prom-label-proxy/0.log"
Jul 02 13:06:36.468921 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:36.468884    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6f21957-0de2-44f3-9625-5e208e2bb4be/init-config-reloader/0.log"
Jul 02 13:06:36.654952 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:36.654883    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-m76wm_33aa4b59-fdea-4a0e-8b70-76b57da84a95/cluster-monitoring-operator/0.log"
Jul 02 13:06:37.393347 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:37.393319    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqzrs_a539811f-cf06-446c-b295-14b8f63ea5ed/node-exporter/0.log"
Jul 02 13:06:37.426305 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:37.426279    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqzrs_a539811f-cf06-446c-b295-14b8f63ea5ed/kube-rbac-proxy/0.log"
Jul 02 13:06:37.462590 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:37.462568    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqzrs_a539811f-cf06-446c-b295-14b8f63ea5ed/init-textfile/0.log"
Jul 02 13:06:38.759714 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:38.759681    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cc4f64768-8d7dt_7982a147-27b6-4a68-bcd8-46df1943ba9f/telemeter-client/0.log"
Jul 02 13:06:38.799158 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:38.799128    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cc4f64768-8d7dt_7982a147-27b6-4a68-bcd8-46df1943ba9f/reload/0.log"
Jul 02 13:06:38.851790 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:38.851764    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cc4f64768-8d7dt_7982a147-27b6-4a68-bcd8-46df1943ba9f/kube-rbac-proxy/0.log"
Jul 02 13:06:39.786672 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.786636    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"]
Jul 02 13:06:39.787144 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.786985    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="storage-initializer"
Jul 02 13:06:39.787144 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.786997    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="storage-initializer"
Jul 02 13:06:39.787144 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.787006    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="kserve-container"
Jul 02 13:06:39.787144 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.787012    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="kserve-container"
Jul 02 13:06:39.787144 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.787019    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="kube-rbac-proxy"
Jul 02 13:06:39.787144 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.787025    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="kube-rbac-proxy"
Jul 02 13:06:39.787144 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.787090    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="kserve-container"
Jul 02 13:06:39.787144 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.787099    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="243651c4-4b41-4814-8e89-c85de903be78" containerName="kube-rbac-proxy"
Jul 02 13:06:39.790021 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.789990    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.792922 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.792903    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-42fsq\"/\"default-dockercfg-2p2v8\""
Jul 02 13:06:39.794104 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.794089    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-42fsq\"/\"kube-root-ca.crt\""
Jul 02 13:06:39.794159 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.794101    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-42fsq\"/\"openshift-service-ca.crt\""
Jul 02 13:06:39.809812 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.809788    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"]
Jul 02 13:06:39.844207 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.844169    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-lib-modules\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.844207 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.844204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-podres\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.844431 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.844257    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-sys\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.844431 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.844319    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-proc\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.844431 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.844387    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj9l7\" (UniqueName: \"kubernetes.io/projected/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-kube-api-access-sj9l7\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.944794 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.944761    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-proc\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.944966 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.944813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj9l7\" (UniqueName: \"kubernetes.io/projected/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-kube-api-access-sj9l7\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.944966 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.944877    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-proc\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.944966 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.944886    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-lib-modules\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.944966 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.944918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-podres\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.944966 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.944939    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-sys\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.945180 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.945004    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-sys\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.945180 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.945013    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-lib-modules\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.945180 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.945043    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-podres\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:39.953546 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:39.953520    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj9l7\" (UniqueName: \"kubernetes.io/projected/79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d-kube-api-access-sj9l7\") pod \"perf-node-gather-daemonset-r7xxg\" (UID: \"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d\") " pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:40.099783 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:40.099694    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:40.246639 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:40.246606    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-8rg49_34416683-a3ce-4dbd-9b1b-b6f1d781b90a/networking-console-plugin/0.log"
Jul 02 13:06:40.250479 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:40.250455    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"]
Jul 02 13:06:40.253671 ip-10-0-143-149 kubenswrapper[2564]: W0702 13:06:40.253638    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod79dfb2b1_ce4d_4e11_aa8c_75ebc01c188d.slice/crio-b384e2edb77bf75d29c9cd60bc57c2b979a39eeca18f579e7c3e98619879cf13 WatchSource:0}: Error finding container b384e2edb77bf75d29c9cd60bc57c2b979a39eeca18f579e7c3e98619879cf13: Status 404 returned error can't find the container with id b384e2edb77bf75d29c9cd60bc57c2b979a39eeca18f579e7c3e98619879cf13
Jul 02 13:06:40.938378 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:40.938339    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg" event={"ID":"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d","Type":"ContainerStarted","Data":"82f58e7fbb534253ed1276f1fb930bf1d3f9184b911952cdf75a4efc8ab4cc74"}
Jul 02 13:06:40.938794 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:40.938391    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg" event={"ID":"79dfb2b1-ce4d-4e11-aa8c-75ebc01c188d","Type":"ContainerStarted","Data":"b384e2edb77bf75d29c9cd60bc57c2b979a39eeca18f579e7c3e98619879cf13"}
Jul 02 13:06:40.938794 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:40.938471    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:40.973794 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:40.973745    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg" podStartSLOduration=1.973729945 podStartE2EDuration="1.973729945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:06:40.973275867 +0000 UTC m=+1648.956749549" watchObservedRunningTime="2026-07-02 13:06:40.973729945 +0000 UTC m=+1648.957203628"
Jul 02 13:06:41.340487 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:41.340408    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d8d54b864-gt52k_5fdf7d99-262b-4054-af0c-12524be97fc8/console/0.log"
Jul 02 13:06:43.122861 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:43.122822    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5bz8v_d173e052-416f-46c8-a7ed-b3b779c66c66/dns/0.log"
Jul 02 13:06:43.173828 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:43.173799    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5bz8v_d173e052-416f-46c8-a7ed-b3b779c66c66/kube-rbac-proxy/0.log"
Jul 02 13:06:43.481668 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:43.481640    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lczx4_e6dde057-4214-4246-bea1-a0820ab52a59/dns-node-resolver/0.log"
Jul 02 13:06:44.191079 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:44.191053    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pgvs9_f7c2b6aa-6cb9-4b5b-885c-4be0975036ff/node-ca/0.log"
Jul 02 13:06:45.740103 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:45.740067    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-drvvs_0318470e-cccb-4d26-96b4-3b464de83f56/serve-healthcheck-canary/0.log"
Jul 02 13:06:46.234534 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:46.234503    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-v24km_49a6bea0-1155-4bd8-b8f6-f6b5a5016167/insights-operator/0.log"
Jul 02 13:06:46.235070 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:46.235052    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-v24km_49a6bea0-1155-4bd8-b8f6-f6b5a5016167/insights-operator/1.log"
Jul 02 13:06:46.405209 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:46.405180    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ksn6z_bf13802d-96c8-49d1-b303-ee39802c83f3/kube-rbac-proxy/0.log"
Jul 02 13:06:46.434174 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:46.434150    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ksn6z_bf13802d-96c8-49d1-b303-ee39802c83f3/exporter/0.log"
Jul 02 13:06:46.464693 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:46.464655    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ksn6z_bf13802d-96c8-49d1-b303-ee39802c83f3/extractor/0.log"
Jul 02 13:06:46.952942 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:46.952916    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-42fsq/perf-node-gather-daemonset-r7xxg"
Jul 02 13:06:49.268219 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:49.268190    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-7hskt_6b8e1c50-765f-43fa-b37f-e75bbbd8d879/manager/0.log"
Jul 02 13:06:49.363457 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:49.363433    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-9n5jl_d86752f9-b757-4fd9-a401-b39000847b11/server/0.log"
Jul 02 13:06:49.799058 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:49.798982    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-k8shf_c6e83fc6-1212-41cb-8391-46204f20f25a/seaweedfs-tls-custom/0.log"
Jul 02 13:06:49.836468 ip-10-0-143-149 kubenswrapper[2564]: I0702 13:06:49.836443    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-gd68j_94f32a34-da9d-491d-aeaf-062ce775664d/seaweedfs-tls-serving/0.log"