Jul 06 21:10:37.586017 ip-10-0-137-165 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 06 21:10:37.586028 ip-10-0-137-165 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 06 21:10:37.586035 ip-10-0-137-165 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 06 21:10:37.586296 ip-10-0-137-165 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 06 21:10:47.790173 ip-10-0-137-165 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 06 21:10:47.790194 ip-10-0-137-165 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4b16e48bcc1048a3830c2792e9d3b0b8 --
Jul 06 21:12:42.872556 ip-10-0-137-165 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 21:12:43.647211 ip-10-0-137-165 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 21:12:43.647211 ip-10-0-137-165 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 21:12:43.647211 ip-10-0-137-165 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 21:12:43.647211 ip-10-0-137-165 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 21:12:43.647211 ip-10-0-137-165 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 21:12:43.649199 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.649075    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 21:12:43.652861 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652844    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 21:12:43.652861 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652861    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652865    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652870    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652873    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652877    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652880    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652883    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652887    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652890    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652893    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652896    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652899    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652902    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652905    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652908    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652911    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652914    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652917    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652920    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652924    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 21:12:43.652935 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652927    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652930    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652933    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652936    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652939    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652942    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652945    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652950    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652954    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652958    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652962    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652965    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652968    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652971    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652974    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652977    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652980    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652982    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652985    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 21:12:43.653450 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652988    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652990    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652993    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652996    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.652998    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653001    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653004    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653006    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653009    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653012    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653015    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653017    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653021    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653023    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653026    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653030    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653033    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653036    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653039    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 21:12:43.653927 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653041    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653046    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653049    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653052    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653056    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653059    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653062    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653065    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653069    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653072    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653075    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653078    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653081    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653084    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653087    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653091    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653096    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653100    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653103    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 21:12:43.654425 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653107    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653110    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653113    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653116    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653118    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653122    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653124    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653127    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653130    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653134    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653137    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653140    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653143    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653146    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653148    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653151    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653154    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653157    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653160    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653162    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 21:12:43.654915 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653165    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653935    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653942    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653945    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653948    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653951    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653959    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653962    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653965    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653968    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653971    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653974    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653977    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653979    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653982    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653985    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653988    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653991    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653994    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.653996    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 21:12:43.655468 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654000    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654004    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654007    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654010    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654013    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654015    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654018    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654021    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654024    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654026    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654029    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654032    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654035    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654038    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654040    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654043    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654046    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654048    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654051    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654054    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 21:12:43.655988 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654057    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654060    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654062    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654065    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654068    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654070    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654073    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654076    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654078    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654081    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654083    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654086    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654089    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654095    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654099    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654102    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654105    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654108    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654111    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654114    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 21:12:43.656530 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654117    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654120    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654123    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654126    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654129    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654132    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654135    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654137    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654140    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654143    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654145    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654148    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654151    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654154    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654157    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654159    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654162    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654165    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654169    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654172    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 21:12:43.657030 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654175    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654178    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654180    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654183    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654186    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654189    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654192    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654196    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654198    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654201    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654204    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654207    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654210    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654212    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654217    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654221    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654224    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654227    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654229    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 21:12:43.657569 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654232    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654645    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654650    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654654    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654656    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654660    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654663    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654666    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654668    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654672    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654675    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654678    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654681    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654684    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654687    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654690    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654692    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654695    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654698    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654701    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 21:12:43.658053 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654704    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654707    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654710    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654713    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654715    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654718    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654721    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654724    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654726    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654729    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654732    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654734    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654737    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654741    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654743    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654746    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654749    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654751    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654755    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654758    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 21:12:43.658614 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654760    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654764    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654766    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654769    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654772    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654774    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654777    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654780    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654783    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654785    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654789    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654791    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654794    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654799    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654802    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654805    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654808    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654812    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654815    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 21:12:43.659109 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654818    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654821    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654824    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654826    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654829    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654832    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654834    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654837    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654840    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654842    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654848    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654851    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654855    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654858    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654864    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654867    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654870    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654873    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654876    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 21:12:43.659636 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654879    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654882    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654885    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654889    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654892    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654895    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654898    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654901    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654903    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654906    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654909    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654911    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654914    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654917    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654920    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654922    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654925    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654928    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654931    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 21:12:43.660141 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654934    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654937    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.654939    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655018    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655030    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655037    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655042    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655046    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655052    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655057    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655064    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655067    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655070    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655074    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655077    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655080    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655083    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655086    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655089    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655093    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655096    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655100    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655103    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 21:12:43.660657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655106    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655110    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655113    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655117    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655120    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655123    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655127    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655130    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655133    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655136    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655139    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655142    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655146    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655149    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655153    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655156    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655160    2581 flags.go:64] FLAG: --enable-server="true"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655165    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655170    2581 flags.go:64] FLAG: --event-burst="100"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655173    2581 flags.go:64] FLAG: --event-qps="50"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655176    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655180    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655183    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655187    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 21:12:43.661281 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655190    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655193    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655196    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655199    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655202    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655205    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655208    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655211    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655214    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655217    2581 flags.go:64] FLAG: --feature-gates=""
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655221    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655225    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655228    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655231    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655234    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655237    2581 flags.go:64] FLAG: --help="false"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655240    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-137-165.ec2.internal"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655243    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655247    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655250    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655253    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655256    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655260    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 21:12:43.661957 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655263    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655267    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655272    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655275    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655279    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655283    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655286    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655289    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655292    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655295    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655298    2581 flags.go:64] FLAG: --lock-file=""
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655314    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655318    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655321    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655327    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655330    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655333    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655336    2581 flags.go:64] FLAG: --logging-format="text"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655339    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655343    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655346    2581 flags.go:64] FLAG: --manifest-url=""
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655349    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655354    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655357    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655361    2581 flags.go:64] FLAG: --max-pods="110"
Jul 06 21:12:43.662621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655365    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655368    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655371    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655374    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655378    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655381    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655384    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655392    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655395    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655402    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655405    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655409    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655415    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655418    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655421    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655424    2581 flags.go:64] FLAG: --port="10250"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655427    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655431    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04b97b36c1cb9e89f"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655434    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655437    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655440    2581 flags.go:64] FLAG: --register-node="true"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655444    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655447    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655450    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 06 21:12:43.663532 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655453    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655456    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655460    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655463    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655466    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655469    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655473    2581 flags.go:64] FLAG: --runonce="false"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655476    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655479    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655483    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655486    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655489    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655492    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655495    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655498    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655503    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655506    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655511    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655514    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655517    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655522    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655525    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655531    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655535    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655537    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 21:12:43.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655542    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655545    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655548    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655552    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655555    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655558    2581 flags.go:64] FLAG: --v="2"
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655562    2581 flags.go:64] FLAG: --version="false"
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655566    2581 flags.go:64] FLAG: --vmodule=""
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655570    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.655574    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655691    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655695    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655698    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655701    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655704    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655707    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655710    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655713    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655716    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655719    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655722    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655725    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 21:12:43.665068 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655731    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655733    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655738    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655741    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655743    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655746    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655750    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655753    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655756    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655758    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655773    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655776    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655779    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655782    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655785    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655787    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655792    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655796    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655799    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655802    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 21:12:43.665703 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655805    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655808    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655811    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655814    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655817    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655819    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655822    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655825    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655828    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655832    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655836    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655840    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655846    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655849    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655854    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655857    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655860    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655863    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655866    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 21:12:43.666229 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655870    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655873    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655876    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655879    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655882    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655885    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655887    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655890    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655893    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655896    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655898    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655901    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655904    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655907    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655909    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655912    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655915    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655918    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655921    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655924    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 21:12:43.666749 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655927    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655930    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655932    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655935    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655938    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655942    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655945    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655949    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655951    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655954    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655957    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655960    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655963    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655966    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655969    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655972    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655975    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655977    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655980    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655983    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 21:12:43.667278 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655986    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655989    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655992    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655995    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.655998    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.656001    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.656004    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.656007    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.656013    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.666425    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.666443    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.666465    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.666474    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666536    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 21:12:43.667822 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666542    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666546    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666550    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666553    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666556    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666560    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666563    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666566    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666569    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666571    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666574    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666577    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666580    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666583    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666587    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666590    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666593    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666596    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666599    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666602    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 21:12:43.668203 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666604    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666607    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666610    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666613    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666616    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666618    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666621    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666625    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666629    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666632    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666634    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666637    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666640    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666643    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666645    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666648    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666651    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666653    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666656    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 21:12:43.668849 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666659    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666662    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666664    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666667    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666670    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666672    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666675    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666679    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666681    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666684    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666687    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666690    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666693    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666695    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666698    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666701    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666703    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666707    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666709    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666712    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 21:12:43.669354 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666715    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666718    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666721    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666724    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666727    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666730    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666733    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666735    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666738    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666741    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666744    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666746    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666749    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666754    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666758    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666761    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666766    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666771    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666774    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 21:12:43.669872 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666778    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666781    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666785    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666789    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666792    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666795    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666798    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666802    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666805    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666808    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666811    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666814    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666817    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666820    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666823    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666826    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666830    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666833    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666835    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 21:12:43.670388 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666838    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.666843    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666981    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666987    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666992    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666996    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.666999    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667002    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667006    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667009    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667012    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667015    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667018    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667021    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 21:12:43.670887 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667025    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667028    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667032    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667035    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667038    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667042    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667045    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667048    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667051    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667054    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667057    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667060    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667063    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667066    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667069    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667072    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667075    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667078    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667080    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 21:12:43.671243 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667083    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667086    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667089    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667091    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667094    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667097    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667100    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667102    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667105    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667132    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667135    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667139    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667143    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667146    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667150    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667152    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667155    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667158    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667160    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667163    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 21:12:43.671780 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667166    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667168    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667171    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667174    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667177    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667179    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667182    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667185    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667188    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667191    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667194    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667197    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667199    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667202    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667205    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667208    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667212    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667215    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667218    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667220    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 21:12:43.672284 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667223    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667226    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667229    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667232    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667235    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667237    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667240    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667246    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667249    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667252    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667255    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667257    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667260    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667263    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667265    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667268    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667271    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667274    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667276    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667279    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 21:12:43.672884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667282    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667285    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667288    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667291    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667294    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667297    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667300    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:43.667321    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.667327    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.667474    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.671325    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.672966    2581 server.go:1019] "Starting client certificate rotation"
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.673080    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 21:12:43.673504 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.673130    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 21:12:43.675549 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.675535    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 21:12:43.676820 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.676797    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 21:12:43.684197 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.684177    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 06 21:12:43.686523 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.686508    2581 log.go:25] "Validated CRI v1 image API"
Jul 06 21:12:43.687304 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.687287    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 21:12:43.691504 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.691485    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ac10d193-95b0-4490-931f-0be15159b939:/dev/nvme0n1p4 c54a326a-76ce-4be9-8b8c-9e32c6a8e8c2:/dev/nvme0n1p3]
Jul 06 21:12:43.691597 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.691504    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 21:12:43.698486 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.698359    2581 manager.go:216] Machine: {Timestamp:2026-07-06 21:12:43.696986417 +0000 UTC m=+0.691085571 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099476 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b57d16c8f4b2ba5c8fb6693db28f1 SystemUUID:ec2b57d1-6c8f-4b2b-a5c8-fb6693db28f1 BootID:4b16e48b-cc10-48a3-830c-2792e9d3b0b8 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4c:c5:5d:f6:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4c:c5:5d:f6:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:c9:37:c5:c4:18 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 21:12:43.698486 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.698474    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 21:12:43.698612 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.698594    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 21:12:43.703059 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.703032    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 21:12:43.703198 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.703063    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-165.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 21:12:43.704607 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.704590    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 21:12:43.705089 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.705077    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 21:12:43.705134 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.705093    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 21:12:43.705134 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.705101    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 21:12:43.705739 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.705727    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 21:12:43.706983 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.706972    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 21:12:43.709809 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.709797    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 21:12:43.709922 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.709913    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 21:12:43.709953 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.709942    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 21:12:43.709953 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.709951    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 21:12:43.710014 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.709968    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 21:12:43.710014 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.709978    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 06 21:12:43.710014 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.709988    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 21:12:43.710343 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.710331    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 21:12:43.710403 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.710361    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 21:12:43.712898 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.712881    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 21:12:43.714875 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.714861    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 21:12:43.714937 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.714893    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 21:12:43.716147 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.716135    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 21:12:43.716203 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.716152    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 21:12:43.716203 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.716159    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 21:12:43.716203 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.716165    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 21:12:43.716203 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.716175    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 21:12:43.716203 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.716181    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 21:12:43.716203 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.716190    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 21:12:43.716203 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.716206    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 21:12:43.716420 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.716214    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 21:12:43.716420 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.716220    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 21:12:43.716420 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.716233    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 21:12:43.716420 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.716239    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 21:12:43.718080 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.718067    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 21:12:43.718080 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.718080    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 21:12:43.721377 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.721353    2581 server.go:1268] "Started kubelet"
Jul 06 21:12:43.721534 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.721507    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 21:12:43.721534 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.721461    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 21:12:43.721642 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.721559    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 21:12:43.722345 ip-10-0-137-165 systemd[1]: Started Kubernetes Kubelet.
Jul 06 21:12:43.723860 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.723605    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-165.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 21:12:43.723860 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.723711    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-165.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 21:12:43.723860 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.723799    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 21:12:43.725086 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.725062    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 21:12:43.726417 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.726400    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 21:12:43.727202 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.727185    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 21:12:43.730882 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.729874    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-165.ec2.internal.18bfcfb751dbd52f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-165.ec2.internal,UID:ip-10-0-137-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-165.ec2.internal,},FirstTimestamp:2026-07-06 21:12:43.721332015 +0000 UTC m=+0.715431169,LastTimestamp:2026-07-06 21:12:43.721332015 +0000 UTC m=+0.715431169,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-165.ec2.internal,}"
Jul 06 21:12:43.732235 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.732218    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 21:12:43.732351 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.732223    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 21:12:43.732351 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.732321    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-165.ec2.internal\" not found"
Jul 06 21:12:43.732857 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.732835    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 21:12:43.732955 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.732841    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 21:12:43.732955 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.732895    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 21:12:43.733070 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.732989    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 21:12:43.733070 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.732999    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 21:12:43.733304 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.733269    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 21:12:43.733378 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.733323    2581 factory.go:55] Registering systemd factory
Jul 06 21:12:43.733378 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.733338    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 06 21:12:43.733651 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.733636    2581 factory.go:153] Registering CRI-O factory
Jul 06 21:12:43.733741 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.733656    2581 factory.go:223] Registration of the crio container factory successfully
Jul 06 21:12:43.733741 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.733679    2581 factory.go:103] Registering Raw factory
Jul 06 21:12:43.733741 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.733694    2581 manager.go:1196] Started watching for new ooms in manager
Jul 06 21:12:43.734836 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.734819    2581 manager.go:349] Starting recovery of all containers
Jul 06 21:12:43.743923 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.743876    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 06 21:12:43.744060 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.744005    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-165.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 06 21:12:43.749815 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.749795    2581 manager.go:354] Recovery completed
Jul 06 21:12:43.754446 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.754432    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 21:12:43.756353 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.756319    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 21:12:43.756436 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.756362    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 21:12:43.756436 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.756383    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientPID"
Jul 06 21:12:43.756580 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.756556    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5x9f4"
Jul 06 21:12:43.757027 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.757011    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 21:12:43.757027 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.757024    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 21:12:43.757132 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.757026    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 21:12:43.757132 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.757103    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 21:12:43.757923 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.757907    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 21:12:43.758018 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.757937    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 21:12:43.758018 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.757964    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientPID"
Jul 06 21:12:43.758409 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.758395    2581 policy_none.go:49] "None policy: Start"
Jul 06 21:12:43.758458 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.758413    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 21:12:43.758458 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.758424    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 21:12:43.758960 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.758878    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-165.ec2.internal.18bfcfb753f20b85  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-165.ec2.internal,UID:ip-10-0-137-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-165.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-165.ec2.internal,},FirstTimestamp:2026-07-06 21:12:43.756342149 +0000 UTC m=+0.750441308,LastTimestamp:2026-07-06 21:12:43.756342149 +0000 UTC m=+0.750441308,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-165.ec2.internal,}"
Jul 06 21:12:43.759512 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.759500    2581 policy_none.go:47] "Start"
Jul 06 21:12:43.764480 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.764459    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5x9f4"
Jul 06 21:12:43.793001 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.792862    2581 manager.go:337] "Starting Device Plugin manager"
Jul 06 21:12:43.793122 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.793028    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 21:12:43.793122 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.793043    2581 server.go:87] "Starting device plugin registration server"
Jul 06 21:12:43.793333 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.793318    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 21:12:43.793396 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.793335    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 21:12:43.793456 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.793422    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 21:12:43.793637 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.793610    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 21:12:43.793637 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.793625    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 21:12:43.794156 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.794136    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 21:12:43.794227 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.794182    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-165.ec2.internal\" not found"
Jul 06 21:12:43.870972 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.870927    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 21:12:43.872214 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.872199    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 21:12:43.872296 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.872236    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 21:12:43.872296 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.872265    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 21:12:43.872416 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.872321    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 21:12:43.875115 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.875093    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 21:12:43.894447 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.894419    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 21:12:43.896677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.896656    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 21:12:43.896756 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.896695    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 21:12:43.896756 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.896710    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientPID"
Jul 06 21:12:43.896756 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.896750    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-165.ec2.internal"
Jul 06 21:12:43.906611 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.906592    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-165.ec2.internal"
Jul 06 21:12:43.906674 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.906618    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-165.ec2.internal\": node \"ip-10-0-137-165.ec2.internal\" not found"
Jul 06 21:12:43.932102 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:43.932077    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-165.ec2.internal\" not found"
Jul 06 21:12:43.972651 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.972613    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-165.ec2.internal"]
Jul 06 21:12:43.972816 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.972689    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 21:12:43.973735 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.973718    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 21:12:43.973840 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.973747    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 21:12:43.973840 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.973758    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientPID"
Jul 06 21:12:43.975346 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.975322    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 21:12:43.975416 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.975401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:43.975472 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.975433    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 21:12:43.976114 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.976097    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 21:12:43.976209 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.976125    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 21:12:43.976209 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.976135    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientPID"
Jul 06 21:12:43.976935 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.976919    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 21:12:43.977012 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.976951    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 21:12:43.977012 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.976962    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientPID"
Jul 06 21:12:43.978333 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.978311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:43.978405 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.978344    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 21:12:43.979059 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.979040    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 21:12:43.979136 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.979068    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 21:12:43.979136 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:43.979080    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeHasSufficientPID"
Jul 06 21:12:44.013320 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:44.013258    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-165.ec2.internal\" not found" node="ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.018078 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:44.018061    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-165.ec2.internal\" not found" node="ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.032571 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:44.032548    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-165.ec2.internal\" not found"
Jul 06 21:12:44.034654 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.034636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eeb79b4a1099d31c932ad94cc67c52f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal\" (UID: \"eeb79b4a1099d31c932ad94cc67c52f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.034707 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.034663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2fa6cec54f4cfb0fa895e12eff069ff2-config\") pod \"kube-apiserver-proxy-ip-10-0-137-165.ec2.internal\" (UID: \"2fa6cec54f4cfb0fa895e12eff069ff2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.034707 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.034682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eeb79b4a1099d31c932ad94cc67c52f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal\" (UID: \"eeb79b4a1099d31c932ad94cc67c52f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.137841 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:44.137791    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-165.ec2.internal\" not found"
Jul 06 21:12:44.137964 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.137895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eeb79b4a1099d31c932ad94cc67c52f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal\" (UID: \"eeb79b4a1099d31c932ad94cc67c52f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.137964 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.137924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eeb79b4a1099d31c932ad94cc67c52f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal\" (UID: \"eeb79b4a1099d31c932ad94cc67c52f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.137964 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.137945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2fa6cec54f4cfb0fa895e12eff069ff2-config\") pod \"kube-apiserver-proxy-ip-10-0-137-165.ec2.internal\" (UID: \"2fa6cec54f4cfb0fa895e12eff069ff2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.138128 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.137979    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2fa6cec54f4cfb0fa895e12eff069ff2-config\") pod \"kube-apiserver-proxy-ip-10-0-137-165.ec2.internal\" (UID: \"2fa6cec54f4cfb0fa895e12eff069ff2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.138128 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.137997    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eeb79b4a1099d31c932ad94cc67c52f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal\" (UID: \"eeb79b4a1099d31c932ad94cc67c52f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.138128 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.137998    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eeb79b4a1099d31c932ad94cc67c52f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal\" (UID: \"eeb79b4a1099d31c932ad94cc67c52f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.238864 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:44.238830    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-165.ec2.internal\" not found"
Jul 06 21:12:44.322456 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.322383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.322456 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.322403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.339213 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:44.339183    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-165.ec2.internal\" not found"
Jul 06 21:12:44.439379 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:44.439343    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-165.ec2.internal\" not found"
Jul 06 21:12:44.539609 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:44.539580    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-165.ec2.internal\" not found"
Jul 06 21:12:44.639972 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:44.639894    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-165.ec2.internal\" not found"
Jul 06 21:12:44.671675 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.671642    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 21:12:44.672141 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.671805    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 21:12:44.740770 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:44.740744    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-165.ec2.internal\" not found"
Jul 06 21:12:44.766818 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.766775    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 21:07:44 +0000 UTC" deadline="2028-03-15 04:39:16.58863263 +0000 UTC"
Jul 06 21:12:44.766818 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.766814    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14815h26m31.821821877s"
Jul 06 21:12:44.775745 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.775713    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 21:12:44.828192 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.828172    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 21:12:44.833371 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.833349    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.838615 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:44.838584    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa6cec54f4cfb0fa895e12eff069ff2.slice/crio-fd472e50a7de2253113086ae39d375553474ffe4b4d98f3e8fd7922a25e39d5d WatchSource:0}: Error finding container fd472e50a7de2253113086ae39d375553474ffe4b4d98f3e8fd7922a25e39d5d: Status 404 returned error can't find the container with id fd472e50a7de2253113086ae39d375553474ffe4b4d98f3e8fd7922a25e39d5d
Jul 06 21:12:44.838921 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:44.838898    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb79b4a1099d31c932ad94cc67c52f9.slice/crio-ba581ddb3d55e692b191f5e837739006eec1c4e85c0615de383d2285d89f50f6 WatchSource:0}: Error finding container ba581ddb3d55e692b191f5e837739006eec1c4e85c0615de383d2285d89f50f6: Status 404 returned error can't find the container with id ba581ddb3d55e692b191f5e837739006eec1c4e85c0615de383d2285d89f50f6
Jul 06 21:12:44.845687 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.845668    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:12:44.846869 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.846852    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 21:12:44.847934 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.847917    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-165.ec2.internal"
Jul 06 21:12:44.857214 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.857197    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 21:12:44.864069 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.864050    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 21:12:44.876589 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.876540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal" event={"ID":"eeb79b4a1099d31c932ad94cc67c52f9","Type":"ContainerStarted","Data":"ba581ddb3d55e692b191f5e837739006eec1c4e85c0615de383d2285d89f50f6"}
Jul 06 21:12:44.877534 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:44.877510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-165.ec2.internal" event={"ID":"2fa6cec54f4cfb0fa895e12eff069ff2","Type":"ContainerStarted","Data":"fd472e50a7de2253113086ae39d375553474ffe4b4d98f3e8fd7922a25e39d5d"}
Jul 06 21:12:45.711349 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.711286    2581 apiserver.go:52] "Watching apiserver"
Jul 06 21:12:45.719307 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.719265    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 21:12:45.719895 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.719870    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc","openshift-image-registry/node-ca-nrrrm","openshift-multus/multus-5thzq","openshift-network-diagnostics/network-check-target-klq8n","openshift-ovn-kubernetes/ovnkube-node-r6p6t","kube-system/konnectivity-agent-x2nkj","kube-system/kube-apiserver-proxy-ip-10-0-137-165.ec2.internal","openshift-cluster-node-tuning-operator/tuned-wpfxb","openshift-dns/node-resolver-vnjbh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal","openshift-multus/multus-additional-cni-plugins-s5lgg","openshift-multus/network-metrics-daemon-7l279","openshift-network-operator/iptables-alerter-7t6dg","kube-system/global-pull-secret-syncer-l4qqt"]
Jul 06 21:12:45.721776 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.721748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7t6dg"
Jul 06 21:12:45.723389 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.723116    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nrrrm"
Jul 06 21:12:45.724745 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.724556    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 21:12:45.724745 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.724723    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m22b6\""
Jul 06 21:12:45.724913 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.724754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.724971 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.724928    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 21:12:45.725518 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.725502    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 21:12:45.725726 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.725697    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 21:12:45.725798 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.725773    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 21:12:45.725856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.725836    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gzx4w\""
Jul 06 21:12:45.726062 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.726043    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 21:12:45.726210 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.726190    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:45.726304 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:45.726277    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:12:45.727250 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.727231    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 21:12:45.727965 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.727633    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 21:12:45.727965 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.727679    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pdmcv\""
Jul 06 21:12:45.727965 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.727756    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 21:12:45.727965 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.727884    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 21:12:45.728215 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.728032    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.729268 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.729249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x2nkj"
Jul 06 21:12:45.730720 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.730685    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 21:12:45.731763 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.731483    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 21:12:45.731763 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.731524    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 21:12:45.731763 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.731611    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 21:12:45.731763 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.731696    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 21:12:45.733136 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.732200    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.733136 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.732278    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dhmc6\""
Jul 06 21:12:45.733136 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.732370    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 21:12:45.733136 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.732865    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 21:12:45.733136 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.733011    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4j7rg\""
Jul 06 21:12:45.733136 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.733097    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 21:12:45.733577 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.733409    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 21:12:45.734703 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.734502    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 21:12:45.734703 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.734671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.735435 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.735407    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 21:12:45.735531 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.735515    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-47d25\""
Jul 06 21:12:45.736069 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.736047    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 21:12:45.736429 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.736408    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vnjbh"
Jul 06 21:12:45.736990 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.736969    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 21:12:45.737171 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.737053    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 21:12:45.737273 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.737056    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ggcj7\""
Jul 06 21:12:45.737962 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.737934    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:45.738055 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:45.737999    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:12:45.738402 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.738382    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 21:12:45.738855 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.738792    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4266f\""
Jul 06 21:12:45.739374 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.739122    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 21:12:45.740935 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.740648    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.742163 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.742143    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:45.742277 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:45.742208    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:12:45.742875 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.742849    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 21:12:45.743040 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.742956    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xkcnq\""
Jul 06 21:12:45.743114 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.743040    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 21:12:45.745661 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.745638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-run\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.745769 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.745675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/737686ff-fa6e-49d2-b9ff-e627b0e82889-tmp\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.745769 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.745702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/831d79b5-c28f-4857-8b33-0b013f995cea-cni-binary-copy\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.745769 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.745747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/831d79b5-c28f-4857-8b33-0b013f995cea-multus-daemon-config\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.745944 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.745774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-run-multus-certs\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.745944 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.745798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-var-lib-openvswitch\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.745944 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.745823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdfc8\" (UniqueName: \"kubernetes.io/projected/737686ff-fa6e-49d2-b9ff-e627b0e82889-kube-api-access-tdfc8\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.745944 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.745857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc05b91f-f482-4025-b0f3-d9bff11999d2-ovn-node-metrics-cert\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.745944 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.745883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cc05b91f-f482-4025-b0f3-d9bff11999d2-ovnkube-script-lib\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.745944 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.745906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-var-lib-kubelet\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.745944 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.745936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-host\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.746289 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.745958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-os-release\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.746289 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-cni-netd\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.746289 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-multus-cni-dir\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.746289 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-cnibin\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.746289 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-run-ovn\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.746289 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-kubernetes\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.746289 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxr8f\" (UniqueName: \"kubernetes.io/projected/25fcb51b-8a18-4ba8-b525-86c727dbabf4-kube-api-access-qxr8f\") pod \"node-ca-nrrrm\" (UID: \"25fcb51b-8a18-4ba8-b525-86c727dbabf4\") " pod="openshift-image-registry/node-ca-nrrrm"
Jul 06 21:12:45.746289 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-hostroot\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-etc-kubernetes\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-kubelet\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-modprobe-d\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-system-cni-dir\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-run-netns\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746415    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-etc-openvswitch\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-device-dir\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-run-netns\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-cni-bin\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-run-k8s-cni-cncf-io\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-var-lib-cni-multus\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-node-log\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.746677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746676    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-log-socket\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-sysctl-d\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-sys-fs\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-sys\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-tuned\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cc05b91f-f482-4025-b0f3-d9bff11999d2-env-overrides\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3c2bf1c-dda7-4db4-9a21-d1230fb66eee-iptables-alerter-script\") pod \"iptables-alerter-7t6dg\" (UID: \"a3c2bf1c-dda7-4db4-9a21-d1230fb66eee\") " pod="openshift-network-operator/iptables-alerter-7t6dg"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-run-systemd\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cc05b91f-f482-4025-b0f3-d9bff11999d2-ovnkube-config\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-socket-dir\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xl42\" (UniqueName: \"kubernetes.io/projected/45b56ec5-de05-4b05-ad34-d6398fccca19-kube-api-access-8xl42\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.746993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-multus-conf-dir\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhdxm\" (UniqueName: \"kubernetes.io/projected/831d79b5-c28f-4857-8b33-0b013f995cea-kube-api-access-fhdxm\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-sysctl-conf\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-lib-modules\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.747271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-slash\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-registration-dir\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-multus-socket-dir-parent\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52nzc\" (UniqueName: \"kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc\") pod \"network-check-target-klq8n\" (UID: \"3f33c872-374e-4583-89da-6457f198a950\") " pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-etc-selinux\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-sysconfig\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-var-lib-kubelet\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3c2bf1c-dda7-4db4-9a21-d1230fb66eee-host-slash\") pod \"iptables-alerter-7t6dg\" (UID: \"a3c2bf1c-dda7-4db4-9a21-d1230fb66eee\") " pod="openshift-network-operator/iptables-alerter-7t6dg"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747357    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxch\" (UniqueName: \"kubernetes.io/projected/cc05b91f-f482-4025-b0f3-d9bff11999d2-kube-api-access-vjxch\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-var-lib-cni-bin\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69673591-604c-4157-9715-fcd796eaf2a8-agent-certs\") pod \"konnectivity-agent-x2nkj\" (UID: \"69673591-604c-4157-9715-fcd796eaf2a8\") " pod="kube-system/konnectivity-agent-x2nkj"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69673591-604c-4157-9715-fcd796eaf2a8-konnectivity-ca\") pod \"konnectivity-agent-x2nkj\" (UID: \"69673591-604c-4157-9715-fcd796eaf2a8\") " pod="kube-system/konnectivity-agent-x2nkj"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hpm8\" (UniqueName: \"kubernetes.io/projected/a3c2bf1c-dda7-4db4-9a21-d1230fb66eee-kube-api-access-8hpm8\") pod \"iptables-alerter-7t6dg\" (UID: \"a3c2bf1c-dda7-4db4-9a21-d1230fb66eee\") " pod="openshift-network-operator/iptables-alerter-7t6dg"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-run-openvswitch\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.748194 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-systemd\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.749146 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25fcb51b-8a18-4ba8-b525-86c727dbabf4-host\") pod \"node-ca-nrrrm\" (UID: \"25fcb51b-8a18-4ba8-b525-86c727dbabf4\") " pod="openshift-image-registry/node-ca-nrrrm"
Jul 06 21:12:45.749146 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/25fcb51b-8a18-4ba8-b525-86c727dbabf4-serviceca\") pod \"node-ca-nrrrm\" (UID: \"25fcb51b-8a18-4ba8-b525-86c727dbabf4\") " pod="openshift-image-registry/node-ca-nrrrm"
Jul 06 21:12:45.749146 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.747763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-systemd-units\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.749146 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.748144    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 21:12:45.770713 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.770691    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9np6p"
Jul 06 21:12:45.779170 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.779142    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9np6p"
Jul 06 21:12:45.833910 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.833882    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 21:12:45.848909 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.848879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3c2bf1c-dda7-4db4-9a21-d1230fb66eee-iptables-alerter-script\") pod \"iptables-alerter-7t6dg\" (UID: \"a3c2bf1c-dda7-4db4-9a21-d1230fb66eee\") " pod="openshift-network-operator/iptables-alerter-7t6dg"
Jul 06 21:12:45.849078 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.848922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-run-systemd\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.849078 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.848947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cc05b91f-f482-4025-b0f3-d9bff11999d2-ovnkube-config\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.849078 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.848987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfa5cd09-70f6-4a62-965e-647b129b69be-cni-binary-copy\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.849078 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zpt9\" (UniqueName: \"kubernetes.io/projected/75a6daab-e23d-43e9-ab04-d52574901b8e-kube-api-access-8zpt9\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:45.849078 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-socket-dir\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.849078 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xl42\" (UniqueName: \"kubernetes.io/projected/45b56ec5-de05-4b05-ad34-d6398fccca19-kube-api-access-8xl42\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.849436 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-multus-conf-dir\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.849436 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhdxm\" (UniqueName: \"kubernetes.io/projected/831d79b5-c28f-4857-8b33-0b013f995cea-kube-api-access-fhdxm\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.849436 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-sysctl-conf\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.849436 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-lib-modules\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.849436 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-slash\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.849436 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-registration-dir\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.849436 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-multus-socket-dir-parent\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.849436 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nzc\" (UniqueName: \"kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc\") pod \"network-check-target-klq8n\" (UID: \"3f33c872-374e-4583-89da-6457f198a950\") " pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:45.849436 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:45.849436 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-etc-selinux\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-sysconfig\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-var-lib-kubelet\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3c2bf1c-dda7-4db4-9a21-d1230fb66eee-host-slash\") pod \"iptables-alerter-7t6dg\" (UID: \"a3c2bf1c-dda7-4db4-9a21-d1230fb66eee\") " pod="openshift-network-operator/iptables-alerter-7t6dg"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxch\" (UniqueName: \"kubernetes.io/projected/cc05b91f-f482-4025-b0f3-d9bff11999d2-kube-api-access-vjxch\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfa5cd09-70f6-4a62-965e-647b129b69be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-var-lib-cni-bin\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69673591-604c-4157-9715-fcd796eaf2a8-agent-certs\") pod \"konnectivity-agent-x2nkj\" (UID: \"69673591-604c-4157-9715-fcd796eaf2a8\") " pod="kube-system/konnectivity-agent-x2nkj"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69673591-604c-4157-9715-fcd796eaf2a8-konnectivity-ca\") pod \"konnectivity-agent-x2nkj\" (UID: \"69673591-604c-4157-9715-fcd796eaf2a8\") " pod="kube-system/konnectivity-agent-x2nkj"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hpm8\" (UniqueName: \"kubernetes.io/projected/a3c2bf1c-dda7-4db4-9a21-d1230fb66eee-kube-api-access-8hpm8\") pod \"iptables-alerter-7t6dg\" (UID: \"a3c2bf1c-dda7-4db4-9a21-d1230fb66eee\") " pod="openshift-network-operator/iptables-alerter-7t6dg"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-run-openvswitch\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/376861d0-1d29-4779-9269-e87fdc65094a-kubelet-config\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-systemd\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25fcb51b-8a18-4ba8-b525-86c727dbabf4-host\") pod \"node-ca-nrrrm\" (UID: \"25fcb51b-8a18-4ba8-b525-86c727dbabf4\") " pod="openshift-image-registry/node-ca-nrrrm"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/25fcb51b-8a18-4ba8-b525-86c727dbabf4-serviceca\") pod \"node-ca-nrrrm\" (UID: \"25fcb51b-8a18-4ba8-b525-86c727dbabf4\") " pod="openshift-image-registry/node-ca-nrrrm"
Jul 06 21:12:45.849987 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-systemd-units\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfa5cd09-70f6-4a62-965e-647b129b69be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.849986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-run\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/737686ff-fa6e-49d2-b9ff-e627b0e82889-tmp\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/831d79b5-c28f-4857-8b33-0b013f995cea-cni-binary-copy\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/831d79b5-c28f-4857-8b33-0b013f995cea-multus-daemon-config\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-run-multus-certs\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-var-lib-openvswitch\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfa5cd09-70f6-4a62-965e-647b129b69be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdfc8\" (UniqueName: \"kubernetes.io/projected/737686ff-fa6e-49d2-b9ff-e627b0e82889-kube-api-access-tdfc8\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc05b91f-f482-4025-b0f3-d9bff11999d2-ovn-node-metrics-cert\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cc05b91f-f482-4025-b0f3-d9bff11999d2-ovnkube-script-lib\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-var-lib-kubelet\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-host\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-os-release\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-cni-netd\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:45.850883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfa5cd09-70f6-4a62-965e-647b129b69be-cnibin\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-multus-cni-dir\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-cnibin\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-run-ovn\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgmq6\" (UniqueName: \"kubernetes.io/projected/dfa5cd09-70f6-4a62-965e-647b129b69be-kube-api-access-xgmq6\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-kubernetes\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxr8f\" (UniqueName: \"kubernetes.io/projected/25fcb51b-8a18-4ba8-b525-86c727dbabf4-kube-api-access-qxr8f\") pod \"node-ca-nrrrm\" (UID: \"25fcb51b-8a18-4ba8-b525-86c727dbabf4\") " pod="openshift-image-registry/node-ca-nrrrm"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-hostroot\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-etc-kubernetes\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-kubelet\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8b1bafa-cee5-4b58-8da4-753d6c0805bd-hosts-file\") pod \"node-resolver-vnjbh\" (UID: \"c8b1bafa-cee5-4b58-8da4-753d6c0805bd\") " pod="openshift-dns/node-resolver-vnjbh"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8b1bafa-cee5-4b58-8da4-753d6c0805bd-tmp-dir\") pod \"node-resolver-vnjbh\" (UID: \"c8b1bafa-cee5-4b58-8da4-753d6c0805bd\") " pod="openshift-dns/node-resolver-vnjbh"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850864    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbtjm\" (UniqueName: \"kubernetes.io/projected/c8b1bafa-cee5-4b58-8da4-753d6c0805bd-kube-api-access-sbtjm\") pod \"node-resolver-vnjbh\" (UID: \"c8b1bafa-cee5-4b58-8da4-753d6c0805bd\") " pod="openshift-dns/node-resolver-vnjbh"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-modprobe-d\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-system-cni-dir\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-run-netns\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.850999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-etc-openvswitch\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.851771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/376861d0-1d29-4779-9269-e87fdc65094a-dbus\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-device-dir\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-run-netns\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-cni-bin\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa5cd09-70f6-4a62-965e-647b129b69be-system-cni-dir\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-run-k8s-cni-cncf-io\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-var-lib-cni-multus\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-node-log\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-log-socket\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfa5cd09-70f6-4a62-965e-647b129b69be-os-release\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-sysctl-d\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-sys-fs\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-sys\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-tuned\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851600    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-run-multus-certs\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.851635    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-run-netns\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.852060    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-run-k8s-cni-cncf-io\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.852604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.852061    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.852120    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-var-lib-openvswitch\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.852110    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-kubelet\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.852164    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-var-lib-cni-multus\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.852182    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-hostroot\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.852211    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-run-systemd\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.852244    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-log-socket\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.852388    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-cni-netd\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.853044    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69673591-604c-4157-9715-fcd796eaf2a8-konnectivity-ca\") pod \"konnectivity-agent-x2nkj\" (UID: \"69673591-604c-4157-9715-fcd796eaf2a8\") " pod="kube-system/konnectivity-agent-x2nkj"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.853116    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-slash\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.853138    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-systemd-units\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.853382    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3c2bf1c-dda7-4db4-9a21-d1230fb66eee-iptables-alerter-script\") pod \"iptables-alerter-7t6dg\" (UID: \"a3c2bf1c-dda7-4db4-9a21-d1230fb66eee\") " pod="openshift-network-operator/iptables-alerter-7t6dg"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.853459    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-run-netns\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.853633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.853576    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-lib-modules\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.854352 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.853733    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/831d79b5-c28f-4857-8b33-0b013f995cea-cni-binary-copy\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.854352 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.853792    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-systemd\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.854352 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.853847    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-multus-conf-dir\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.854352 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.853858    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-device-dir\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.855476    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cc05b91f-f482-4025-b0f3-d9bff11999d2-ovnkube-config\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.855601    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-sysctl-conf\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.855662    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-var-lib-kubelet\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.855783    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-sysconfig\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.855810    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-etc-selinux\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.855891    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-run-ovn\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.855923    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.855976    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-node-log\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.856090    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-sysctl-d\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.856143    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-sys-fs\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.856142    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/831d79b5-c28f-4857-8b33-0b013f995cea-multus-daemon-config\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.856179    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-etc-kubernetes\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.853793    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-etc-openvswitch\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.857033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.856291    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-os-release\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.857044    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cc05b91f-f482-4025-b0f3-d9bff11999d2-ovnkube-script-lib\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.857297    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-sys\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.857562    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-registration-dir\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.857679    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc05b91f-f482-4025-b0f3-d9bff11999d2-ovn-node-metrics-cert\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.857824    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-modprobe-d\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.857922    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-multus-socket-dir-parent\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.857969    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-kubernetes\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858021    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-cnibin\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858060    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3c2bf1c-dda7-4db4-9a21-d1230fb66eee-host-slash\") pod \"iptables-alerter-7t6dg\" (UID: \"a3c2bf1c-dda7-4db4-9a21-d1230fb66eee\") " pod="openshift-network-operator/iptables-alerter-7t6dg"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858104    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-cni-bin\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858153    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-system-cni-dir\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858255    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-socket-dir\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858343    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/45b56ec5-de05-4b05-ad34-d6398fccca19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858393    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-var-lib-kubelet\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858428    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-host-var-lib-cni-bin\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858527    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/831d79b5-c28f-4857-8b33-0b013f995cea-multus-cni-dir\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858568    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.859570 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858607    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc05b91f-f482-4025-b0f3-d9bff11999d2-run-openvswitch\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.860260 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858646    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25fcb51b-8a18-4ba8-b525-86c727dbabf4-host\") pod \"node-ca-nrrrm\" (UID: \"25fcb51b-8a18-4ba8-b525-86c727dbabf4\") " pod="openshift-image-registry/node-ca-nrrrm"
Jul 06 21:12:45.860260 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.858926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cc05b91f-f482-4025-b0f3-d9bff11999d2-env-overrides\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.860260 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.859076    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-run\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.860260 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.859415    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cc05b91f-f482-4025-b0f3-d9bff11999d2-env-overrides\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.860260 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.859780    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/737686ff-fa6e-49d2-b9ff-e627b0e82889-host\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.860260 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.860223    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/25fcb51b-8a18-4ba8-b525-86c727dbabf4-serviceca\") pod \"node-ca-nrrrm\" (UID: \"25fcb51b-8a18-4ba8-b525-86c727dbabf4\") " pod="openshift-image-registry/node-ca-nrrrm"
Jul 06 21:12:45.861682 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.861534    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/737686ff-fa6e-49d2-b9ff-e627b0e82889-tmp\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.862129 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.861906    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/737686ff-fa6e-49d2-b9ff-e627b0e82889-etc-tuned\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.863720 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.863694    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69673591-604c-4157-9715-fcd796eaf2a8-agent-certs\") pod \"konnectivity-agent-x2nkj\" (UID: \"69673591-604c-4157-9715-fcd796eaf2a8\") " pod="kube-system/konnectivity-agent-x2nkj"
Jul 06 21:12:45.864920 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:45.864894    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 21:12:45.865014 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:45.864925    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 21:12:45.865014 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:45.864940    2581 projected.go:196] Error preparing data for projected volume kube-api-access-52nzc for pod openshift-network-diagnostics/network-check-target-klq8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:12:45.865143 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:45.865031    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc podName:3f33c872-374e-4583-89da-6457f198a950 nodeName:}" failed. No retries permitted until 2026-07-06 21:12:46.364994425 +0000 UTC m=+3.359093586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-52nzc" (UniqueName: "kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc") pod "network-check-target-klq8n" (UID: "3f33c872-374e-4583-89da-6457f198a950") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:12:45.866252 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.866227    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhdxm\" (UniqueName: \"kubernetes.io/projected/831d79b5-c28f-4857-8b33-0b013f995cea-kube-api-access-fhdxm\") pod \"multus-5thzq\" (UID: \"831d79b5-c28f-4857-8b33-0b013f995cea\") " pod="openshift-multus/multus-5thzq"
Jul 06 21:12:45.866368 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.866338    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdfc8\" (UniqueName: \"kubernetes.io/projected/737686ff-fa6e-49d2-b9ff-e627b0e82889-kube-api-access-tdfc8\") pod \"tuned-wpfxb\" (UID: \"737686ff-fa6e-49d2-b9ff-e627b0e82889\") " pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:45.866714 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.866691    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hpm8\" (UniqueName: \"kubernetes.io/projected/a3c2bf1c-dda7-4db4-9a21-d1230fb66eee-kube-api-access-8hpm8\") pod \"iptables-alerter-7t6dg\" (UID: \"a3c2bf1c-dda7-4db4-9a21-d1230fb66eee\") " pod="openshift-network-operator/iptables-alerter-7t6dg"
Jul 06 21:12:45.867065 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.867014    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxch\" (UniqueName: \"kubernetes.io/projected/cc05b91f-f482-4025-b0f3-d9bff11999d2-kube-api-access-vjxch\") pod \"ovnkube-node-r6p6t\" (UID: \"cc05b91f-f482-4025-b0f3-d9bff11999d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:45.867678 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.867659    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xl42\" (UniqueName: \"kubernetes.io/projected/45b56ec5-de05-4b05-ad34-d6398fccca19-kube-api-access-8xl42\") pod \"aws-ebs-csi-driver-node-nf8tc\" (UID: \"45b56ec5-de05-4b05-ad34-d6398fccca19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:45.868676 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.868630    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxr8f\" (UniqueName: \"kubernetes.io/projected/25fcb51b-8a18-4ba8-b525-86c727dbabf4-kube-api-access-qxr8f\") pod \"node-ca-nrrrm\" (UID: \"25fcb51b-8a18-4ba8-b525-86c727dbabf4\") " pod="openshift-image-registry/node-ca-nrrrm"
Jul 06 21:12:45.959411 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa5cd09-70f6-4a62-965e-647b129b69be-system-cni-dir\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.959598 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfa5cd09-70f6-4a62-965e-647b129b69be-os-release\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.959598 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfa5cd09-70f6-4a62-965e-647b129b69be-cni-binary-copy\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.959598 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zpt9\" (UniqueName: \"kubernetes.io/projected/75a6daab-e23d-43e9-ab04-d52574901b8e-kube-api-access-8zpt9\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:45.959598 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:45.959598 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959556    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfa5cd09-70f6-4a62-965e-647b129b69be-os-release\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.959830 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfa5cd09-70f6-4a62-965e-647b129b69be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.959830 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/376861d0-1d29-4779-9269-e87fdc65094a-kubelet-config\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:45.959830 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfa5cd09-70f6-4a62-965e-647b129b69be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.959830 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfa5cd09-70f6-4a62-965e-647b129b69be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.959830 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:45.959830 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfa5cd09-70f6-4a62-965e-647b129b69be-cnibin\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.959830 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgmq6\" (UniqueName: \"kubernetes.io/projected/dfa5cd09-70f6-4a62-965e-647b129b69be-kube-api-access-xgmq6\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.959830 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8b1bafa-cee5-4b58-8da4-753d6c0805bd-hosts-file\") pod \"node-resolver-vnjbh\" (UID: \"c8b1bafa-cee5-4b58-8da4-753d6c0805bd\") " pod="openshift-dns/node-resolver-vnjbh"
Jul 06 21:12:45.960175 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8b1bafa-cee5-4b58-8da4-753d6c0805bd-tmp-dir\") pod \"node-resolver-vnjbh\" (UID: \"c8b1bafa-cee5-4b58-8da4-753d6c0805bd\") " pod="openshift-dns/node-resolver-vnjbh"
Jul 06 21:12:45.960175 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbtjm\" (UniqueName: \"kubernetes.io/projected/c8b1bafa-cee5-4b58-8da4-753d6c0805bd-kube-api-access-sbtjm\") pod \"node-resolver-vnjbh\" (UID: \"c8b1bafa-cee5-4b58-8da4-753d6c0805bd\") " pod="openshift-dns/node-resolver-vnjbh"
Jul 06 21:12:45.960175 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/376861d0-1d29-4779-9269-e87fdc65094a-dbus\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:45.960175 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959971    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/376861d0-1d29-4779-9269-e87fdc65094a-dbus\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:45.960175 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:45.960068    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 21:12:45.960175 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:45.960159    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret podName:376861d0-1d29-4779-9269-e87fdc65094a nodeName:}" failed. No retries permitted until 2026-07-06 21:12:46.460122099 +0000 UTC m=+3.454221257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret") pod "global-pull-secret-syncer-l4qqt" (UID: "376861d0-1d29-4779-9269-e87fdc65094a") : object "kube-system"/"original-pull-secret" not registered
Jul 06 21:12:45.960531 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.959483    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa5cd09-70f6-4a62-965e-647b129b69be-system-cni-dir\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.960531 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.960402    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfa5cd09-70f6-4a62-965e-647b129b69be-cnibin\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.960531 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:45.960448    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:12:45.960531 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.960483    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfa5cd09-70f6-4a62-965e-647b129b69be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.960531 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:45.960516    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs podName:75a6daab-e23d-43e9-ab04-d52574901b8e nodeName:}" failed. No retries permitted until 2026-07-06 21:12:46.460498262 +0000 UTC m=+3.454597420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs") pod "network-metrics-daemon-7l279" (UID: "75a6daab-e23d-43e9-ab04-d52574901b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:12:45.960531 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.960529    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfa5cd09-70f6-4a62-965e-647b129b69be-cni-binary-copy\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.960832 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.960529    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfa5cd09-70f6-4a62-965e-647b129b69be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.960832 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.960568    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8b1bafa-cee5-4b58-8da4-753d6c0805bd-hosts-file\") pod \"node-resolver-vnjbh\" (UID: \"c8b1bafa-cee5-4b58-8da4-753d6c0805bd\") " pod="openshift-dns/node-resolver-vnjbh"
Jul 06 21:12:45.960832 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.960581    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/376861d0-1d29-4779-9269-e87fdc65094a-kubelet-config\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:45.960832 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.960787    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8b1bafa-cee5-4b58-8da4-753d6c0805bd-tmp-dir\") pod \"node-resolver-vnjbh\" (UID: \"c8b1bafa-cee5-4b58-8da4-753d6c0805bd\") " pod="openshift-dns/node-resolver-vnjbh"
Jul 06 21:12:45.961085 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.961055    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfa5cd09-70f6-4a62-965e-647b129b69be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.968773 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.968714    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zpt9\" (UniqueName: \"kubernetes.io/projected/75a6daab-e23d-43e9-ab04-d52574901b8e-kube-api-access-8zpt9\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:45.972148 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.972122    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgmq6\" (UniqueName: \"kubernetes.io/projected/dfa5cd09-70f6-4a62-965e-647b129b69be-kube-api-access-xgmq6\") pod \"multus-additional-cni-plugins-s5lgg\" (UID: \"dfa5cd09-70f6-4a62-965e-647b129b69be\") " pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:45.972293 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:45.972191    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbtjm\" (UniqueName: \"kubernetes.io/projected/c8b1bafa-cee5-4b58-8da4-753d6c0805bd-kube-api-access-sbtjm\") pod \"node-resolver-vnjbh\" (UID: \"c8b1bafa-cee5-4b58-8da4-753d6c0805bd\") " pod="openshift-dns/node-resolver-vnjbh"
Jul 06 21:12:46.044687 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.044655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7t6dg"
Jul 06 21:12:46.050771 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.050747    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nrrrm"
Jul 06 21:12:46.060815 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.060784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5thzq"
Jul 06 21:12:46.068024 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.068006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:12:46.074432 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.074412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x2nkj"
Jul 06 21:12:46.082412 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.082394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc"
Jul 06 21:12:46.089604 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.089585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wpfxb"
Jul 06 21:12:46.097394 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.097374    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vnjbh"
Jul 06 21:12:46.104178 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.104156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s5lgg"
Jul 06 21:12:46.233456 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.233387    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 21:12:46.465621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.465573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:46.465825 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.465649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nzc\" (UniqueName: \"kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc\") pod \"network-check-target-klq8n\" (UID: \"3f33c872-374e-4583-89da-6457f198a950\") " pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:46.465825 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.465677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:46.465825 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:46.465762    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 21:12:46.465825 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:46.465795    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:12:46.465825 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:46.465818    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 21:12:46.466026 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:46.465833    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret podName:376861d0-1d29-4779-9269-e87fdc65094a nodeName:}" failed. No retries permitted until 2026-07-06 21:12:47.465818115 +0000 UTC m=+4.459917255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret") pod "global-pull-secret-syncer-l4qqt" (UID: "376861d0-1d29-4779-9269-e87fdc65094a") : object "kube-system"/"original-pull-secret" not registered
Jul 06 21:12:46.466026 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:46.465841    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 21:12:46.466026 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:46.465850    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs podName:75a6daab-e23d-43e9-ab04-d52574901b8e nodeName:}" failed. No retries permitted until 2026-07-06 21:12:47.465842163 +0000 UTC m=+4.459941303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs") pod "network-metrics-daemon-7l279" (UID: "75a6daab-e23d-43e9-ab04-d52574901b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:12:46.466026 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:46.465853    2581 projected.go:196] Error preparing data for projected volume kube-api-access-52nzc for pod openshift-network-diagnostics/network-check-target-klq8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:12:46.466026 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:46.465892    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc podName:3f33c872-374e-4583-89da-6457f198a950 nodeName:}" failed. No retries permitted until 2026-07-06 21:12:47.465880273 +0000 UTC m=+4.459979442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-52nzc" (UniqueName: "kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc") pod "network-check-target-klq8n" (UID: "3f33c872-374e-4583-89da-6457f198a950") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:12:46.781860 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.781808    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 21:07:46 +0000 UTC" deadline="2028-03-17 02:20:11.599987792 +0000 UTC"
Jul 06 21:12:46.781860 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:46.781853    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14861h7m24.818137105s"
Jul 06 21:12:47.023234 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:47.023201    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b1bafa_cee5_4b58_8da4_753d6c0805bd.slice/crio-c0625e5151655207925728271d08981c4d3ba2b5972bc6e61c7e8c2c795e32fd WatchSource:0}: Error finding container c0625e5151655207925728271d08981c4d3ba2b5972bc6e61c7e8c2c795e32fd: Status 404 returned error can't find the container with id c0625e5151655207925728271d08981c4d3ba2b5972bc6e61c7e8c2c795e32fd
Jul 06 21:12:47.024816 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:47.024653    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fcb51b_8a18_4ba8_b525_86c727dbabf4.slice/crio-e7b2ee9c26636c63451102957b4f59d0daa5ce1865b5e4e2e5332353acfec71f WatchSource:0}: Error finding container e7b2ee9c26636c63451102957b4f59d0daa5ce1865b5e4e2e5332353acfec71f: Status 404 returned error can't find the container with id e7b2ee9c26636c63451102957b4f59d0daa5ce1865b5e4e2e5332353acfec71f
Jul 06 21:12:47.027941 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:47.027918    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa5cd09_70f6_4a62_965e_647b129b69be.slice/crio-9d13ca57fbc2f553aff981dea26af37f88b08f4c84c202489727c45dcec99d6c WatchSource:0}: Error finding container 9d13ca57fbc2f553aff981dea26af37f88b08f4c84c202489727c45dcec99d6c: Status 404 returned error can't find the container with id 9d13ca57fbc2f553aff981dea26af37f88b08f4c84c202489727c45dcec99d6c
Jul 06 21:12:47.028993 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:47.028960    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c2bf1c_dda7_4db4_9a21_d1230fb66eee.slice/crio-fbec1967b1c4e9198eddee20f3948cc2b8937b6fd873dcb09669df393d94eaa8 WatchSource:0}: Error finding container fbec1967b1c4e9198eddee20f3948cc2b8937b6fd873dcb09669df393d94eaa8: Status 404 returned error can't find the container with id fbec1967b1c4e9198eddee20f3948cc2b8937b6fd873dcb09669df393d94eaa8
Jul 06 21:12:47.030455 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:47.030427    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737686ff_fa6e_49d2_b9ff_e627b0e82889.slice/crio-5e20537f0a26a143cd2651f4187d5b6f676e13c3e4051821c065213bf24f5762 WatchSource:0}: Error finding container 5e20537f0a26a143cd2651f4187d5b6f676e13c3e4051821c065213bf24f5762: Status 404 returned error can't find the container with id 5e20537f0a26a143cd2651f4187d5b6f676e13c3e4051821c065213bf24f5762
Jul 06 21:12:47.032110 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:12:47.032008    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831d79b5_c28f_4857_8b33_0b013f995cea.slice/crio-75fbb7ed001d36023299634fb884effb5a82d2f667cbf3cff870cd248cfef5c0 WatchSource:0}: Error finding container 75fbb7ed001d36023299634fb884effb5a82d2f667cbf3cff870cd248cfef5c0: Status 404 returned error can't find the container with id 75fbb7ed001d36023299634fb884effb5a82d2f667cbf3cff870cd248cfef5c0
Jul 06 21:12:47.475367 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.474414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:47.475367 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.474750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nzc\" (UniqueName: \"kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc\") pod \"network-check-target-klq8n\" (UID: \"3f33c872-374e-4583-89da-6457f198a950\") " pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:47.475367 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.474788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:47.475367 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:47.474920    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:12:47.475367 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:47.474986    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs podName:75a6daab-e23d-43e9-ab04-d52574901b8e nodeName:}" failed. No retries permitted until 2026-07-06 21:12:49.474967645 +0000 UTC m=+6.469066805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs") pod "network-metrics-daemon-7l279" (UID: "75a6daab-e23d-43e9-ab04-d52574901b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:12:47.475367 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:47.475023    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 21:12:47.475367 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:47.475065    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret podName:376861d0-1d29-4779-9269-e87fdc65094a nodeName:}" failed. No retries permitted until 2026-07-06 21:12:49.475052238 +0000 UTC m=+6.469151396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret") pod "global-pull-secret-syncer-l4qqt" (UID: "376861d0-1d29-4779-9269-e87fdc65094a") : object "kube-system"/"original-pull-secret" not registered
Jul 06 21:12:47.475367 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:47.475175    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 21:12:47.475367 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:47.475189    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 21:12:47.475367 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:47.475203    2581 projected.go:196] Error preparing data for projected volume kube-api-access-52nzc for pod openshift-network-diagnostics/network-check-target-klq8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:12:47.475367 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:47.475238    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc podName:3f33c872-374e-4583-89da-6457f198a950 nodeName:}" failed. No retries permitted until 2026-07-06 21:12:49.475226792 +0000 UTC m=+6.469325949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-52nzc" (UniqueName: "kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc") pod "network-check-target-klq8n" (UID: "3f33c872-374e-4583-89da-6457f198a950") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:12:47.782477 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.782428    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 21:07:46 +0000 UTC" deadline="2028-02-10 19:04:15.068870147 +0000 UTC"
Jul 06 21:12:47.782477 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.782475    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14013h51m27.286400716s"
Jul 06 21:12:47.876004 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.875922    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:47.876167 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:47.876087    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:12:47.876739 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.876550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:47.876739 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:47.876654    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:12:47.876739 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.876728    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:47.876956 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:47.876809    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:12:47.896255 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.896216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7t6dg" event={"ID":"a3c2bf1c-dda7-4db4-9a21-d1230fb66eee","Type":"ContainerStarted","Data":"fbec1967b1c4e9198eddee20f3948cc2b8937b6fd873dcb09669df393d94eaa8"}
Jul 06 21:12:47.899609 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.899576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5lgg" event={"ID":"dfa5cd09-70f6-4a62-965e-647b129b69be","Type":"ContainerStarted","Data":"9d13ca57fbc2f553aff981dea26af37f88b08f4c84c202489727c45dcec99d6c"}
Jul 06 21:12:47.911942 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.911856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vnjbh" event={"ID":"c8b1bafa-cee5-4b58-8da4-753d6c0805bd","Type":"ContainerStarted","Data":"c0625e5151655207925728271d08981c4d3ba2b5972bc6e61c7e8c2c795e32fd"}
Jul 06 21:12:47.922903 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.922815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc" event={"ID":"45b56ec5-de05-4b05-ad34-d6398fccca19","Type":"ContainerStarted","Data":"79bb4fa3aadb1baa284a16f5077072b4545ed7912468e304e6fa6f743245165e"}
Jul 06 21:12:47.940786 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.940704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5thzq" event={"ID":"831d79b5-c28f-4857-8b33-0b013f995cea","Type":"ContainerStarted","Data":"75fbb7ed001d36023299634fb884effb5a82d2f667cbf3cff870cd248cfef5c0"}
Jul 06 21:12:47.946749 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.946674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nrrrm" event={"ID":"25fcb51b-8a18-4ba8-b525-86c727dbabf4","Type":"ContainerStarted","Data":"e7b2ee9c26636c63451102957b4f59d0daa5ce1865b5e4e2e5332353acfec71f"}
Jul 06 21:12:47.953808 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.953708    2581 generic.go:356] "Generic (PLEG): container finished" podID="eeb79b4a1099d31c932ad94cc67c52f9" containerID="1b266af6b10e146deffb6131d6f66d7324fb2147a0ff038e3e12da3dcb260f7b" exitCode=0
Jul 06 21:12:47.953926 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.953854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal" event={"ID":"eeb79b4a1099d31c932ad94cc67c52f9","Type":"ContainerDied","Data":"1b266af6b10e146deffb6131d6f66d7324fb2147a0ff038e3e12da3dcb260f7b"}
Jul 06 21:12:47.969545 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.969426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-165.ec2.internal" event={"ID":"2fa6cec54f4cfb0fa895e12eff069ff2","Type":"ContainerStarted","Data":"f499e09dfbb8797db5f4a54c60b3752636e4658cd44552167312ba2629121e51"}
Jul 06 21:12:47.979688 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.979566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t" event={"ID":"cc05b91f-f482-4025-b0f3-d9bff11999d2","Type":"ContainerStarted","Data":"026327f2c6fd9b91159860c883d34946e78792f07eeabddef8641acc5c892366"}
Jul 06 21:12:47.998824 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:47.998722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x2nkj" event={"ID":"69673591-604c-4157-9715-fcd796eaf2a8","Type":"ContainerStarted","Data":"1f0afe861555f31e1941d15d032771e5becdd4e7c72a4144e973147398903b58"}
Jul 06 21:12:48.027332 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:48.027281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wpfxb" event={"ID":"737686ff-fa6e-49d2-b9ff-e627b0e82889","Type":"ContainerStarted","Data":"5e20537f0a26a143cd2651f4187d5b6f676e13c3e4051821c065213bf24f5762"}
Jul 06 21:12:49.053426 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:49.053378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal" event={"ID":"eeb79b4a1099d31c932ad94cc67c52f9","Type":"ContainerStarted","Data":"7ff1b861e19de6aaaf836570394ff6f6a9a4d6b98bd0e2667ebc38ed271fdb5b"}
Jul 06 21:12:49.071931 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:49.070690    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-165.ec2.internal" podStartSLOduration=4.070668618 podStartE2EDuration="4.070668618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:12:49.070508839 +0000 UTC m=+6.064608004" watchObservedRunningTime="2026-07-06 21:12:49.070668618 +0000 UTC m=+6.064767783"
Jul 06 21:12:49.071931 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:49.071771    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-165.ec2.internal" podStartSLOduration=4.071758359 podStartE2EDuration="4.071758359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:12:47.997957793 +0000 UTC m=+4.992056958" watchObservedRunningTime="2026-07-06 21:12:49.071758359 +0000 UTC m=+6.065857528"
Jul 06 21:12:49.490543 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:49.490348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:49.490543 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:49.490442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nzc\" (UniqueName: \"kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc\") pod \"network-check-target-klq8n\" (UID: \"3f33c872-374e-4583-89da-6457f198a950\") " pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:49.490543 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:49.490470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:49.490839 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:49.490598    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:12:49.490839 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:49.490669    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs podName:75a6daab-e23d-43e9-ab04-d52574901b8e nodeName:}" failed. No retries permitted until 2026-07-06 21:12:53.49064711 +0000 UTC m=+10.484746249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs") pod "network-metrics-daemon-7l279" (UID: "75a6daab-e23d-43e9-ab04-d52574901b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:12:49.491192 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:49.491086    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 21:12:49.491192 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:49.491144    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret podName:376861d0-1d29-4779-9269-e87fdc65094a nodeName:}" failed. No retries permitted until 2026-07-06 21:12:53.491127322 +0000 UTC m=+10.485226478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret") pod "global-pull-secret-syncer-l4qqt" (UID: "376861d0-1d29-4779-9269-e87fdc65094a") : object "kube-system"/"original-pull-secret" not registered
Jul 06 21:12:49.491377 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:49.491229    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 21:12:49.491377 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:49.491244    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 21:12:49.491377 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:49.491256    2581 projected.go:196] Error preparing data for projected volume kube-api-access-52nzc for pod openshift-network-diagnostics/network-check-target-klq8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:12:49.491377 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:49.491294    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc podName:3f33c872-374e-4583-89da-6457f198a950 nodeName:}" failed. No retries permitted until 2026-07-06 21:12:53.491281707 +0000 UTC m=+10.485380862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-52nzc" (UniqueName: "kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc") pod "network-check-target-klq8n" (UID: "3f33c872-374e-4583-89da-6457f198a950") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:12:49.877234 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:49.876251    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:49.877234 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:49.876418    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:12:49.877234 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:49.876833    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:49.877234 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:49.876934    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:12:49.877234 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:49.877001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:49.877234 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:49.877082    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:12:51.873330 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:51.873282    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:51.873830 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:51.873284    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:51.873830 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:51.873427    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:12:51.873830 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:51.873444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:51.873830 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:51.873545    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:12:51.873830 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:51.873660    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:12:53.522366 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:53.522328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nzc\" (UniqueName: \"kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc\") pod \"network-check-target-klq8n\" (UID: \"3f33c872-374e-4583-89da-6457f198a950\") " pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:53.522839 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:53.522376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:53.522839 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:53.522421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:53.522839 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:53.522510    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 21:12:53.522839 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:53.522518    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 21:12:53.522839 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:53.522535    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 21:12:53.522839 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:53.522547    2581 projected.go:196] Error preparing data for projected volume kube-api-access-52nzc for pod openshift-network-diagnostics/network-check-target-klq8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:12:53.522839 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:53.522571    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret podName:376861d0-1d29-4779-9269-e87fdc65094a nodeName:}" failed. No retries permitted until 2026-07-06 21:13:01.522552992 +0000 UTC m=+18.516652154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret") pod "global-pull-secret-syncer-l4qqt" (UID: "376861d0-1d29-4779-9269-e87fdc65094a") : object "kube-system"/"original-pull-secret" not registered
Jul 06 21:12:53.522839 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:53.522588    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc podName:3f33c872-374e-4583-89da-6457f198a950 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:01.522578705 +0000 UTC m=+18.516677852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-52nzc" (UniqueName: "kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc") pod "network-check-target-klq8n" (UID: "3f33c872-374e-4583-89da-6457f198a950") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:12:53.522839 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:53.522622    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:12:53.522839 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:53.522651    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs podName:75a6daab-e23d-43e9-ab04-d52574901b8e nodeName:}" failed. No retries permitted until 2026-07-06 21:13:01.522640341 +0000 UTC m=+18.516739490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs") pod "network-metrics-daemon-7l279" (UID: "75a6daab-e23d-43e9-ab04-d52574901b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:12:53.874331 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:53.873545    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:53.874331 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:53.873656    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:12:53.874331 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:53.874016    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:53.874331 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:53.874113    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:12:53.874331 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:53.874164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:53.874331 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:53.874251    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:12:55.873633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:55.873598    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:55.874099 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:55.873747    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:12:55.874534 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:55.874276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:55.874534 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:55.874377    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:12:55.874534 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:55.874432    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:55.874534 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:55.874499    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:12:57.873159 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:57.873018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:57.873159 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:57.873025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:57.873159 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:57.873025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:57.873735 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:57.873196    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:12:57.873735 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:57.873363    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:12:57.873735 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:57.873448    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:12:59.873046 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:59.873005    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:12:59.873516 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:59.873005    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:12:59.873516 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:59.873151    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:12:59.873516 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:12:59.873010    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:12:59.873516 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:59.873224    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:12:59.873516 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:12:59.873344    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:13:01.590351 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:01.590311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nzc\" (UniqueName: \"kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc\") pod \"network-check-target-klq8n\" (UID: \"3f33c872-374e-4583-89da-6457f198a950\") " pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:01.590351 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:01.590352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:01.590819 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:01.590396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:01.590819 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:01.590479    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 21:13:01.590819 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:01.590500    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 21:13:01.590819 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:01.590510    2581 projected.go:196] Error preparing data for projected volume kube-api-access-52nzc for pod openshift-network-diagnostics/network-check-target-klq8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:13:01.590819 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:01.590517    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:13:01.590819 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:01.590517    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 21:13:01.590819 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:01.590557    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc podName:3f33c872-374e-4583-89da-6457f198a950 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:17.590542573 +0000 UTC m=+34.584641727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-52nzc" (UniqueName: "kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc") pod "network-check-target-klq8n" (UID: "3f33c872-374e-4583-89da-6457f198a950") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:13:01.590819 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:01.590596    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs podName:75a6daab-e23d-43e9-ab04-d52574901b8e nodeName:}" failed. No retries permitted until 2026-07-06 21:13:17.590583011 +0000 UTC m=+34.584682148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs") pod "network-metrics-daemon-7l279" (UID: "75a6daab-e23d-43e9-ab04-d52574901b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:13:01.590819 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:01.590607    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret podName:376861d0-1d29-4779-9269-e87fdc65094a nodeName:}" failed. No retries permitted until 2026-07-06 21:13:17.590601154 +0000 UTC m=+34.584700292 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret") pod "global-pull-secret-syncer-l4qqt" (UID: "376861d0-1d29-4779-9269-e87fdc65094a") : object "kube-system"/"original-pull-secret" not registered
Jul 06 21:13:01.872585 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:01.872483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:01.872741 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:01.872484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:01.872741 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:01.872616    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:13:01.872845 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:01.872731    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:13:01.872845 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:01.872492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:01.872845 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:01.872818    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:13:03.873452 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:03.873415    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:03.873933 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:03.873533    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:03.873933 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:03.873542    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:13:03.873933 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:03.873554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:03.873933 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:03.873653    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:13:03.873933 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:03.873743    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:13:05.873430 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:05.873358    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:05.873822 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:05.873492    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:13:05.873822 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:05.873801    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:05.873926 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:05.873898    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:13:05.873982 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:05.873947    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:05.874037 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:05.874009    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:13:06.096710 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.096391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t" event={"ID":"cc05b91f-f482-4025-b0f3-d9bff11999d2","Type":"ContainerStarted","Data":"3f90f4af2bada4c3786bb12ab6c8df5c73aa1c3062fa2078c98dfe3bcfd6a311"}
Jul 06 21:13:06.096840 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.096761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t" event={"ID":"cc05b91f-f482-4025-b0f3-d9bff11999d2","Type":"ContainerStarted","Data":"0651d093ded0f231dd96363e85f2f998929e3a054b2939d2484490c08c603332"}
Jul 06 21:13:06.097855 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.097829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x2nkj" event={"ID":"69673591-604c-4157-9715-fcd796eaf2a8","Type":"ContainerStarted","Data":"ac95b24502129926607134be1a02b0b48407a54bc917a623aa8b8596ff3a7513"}
Jul 06 21:13:06.099113 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.099092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wpfxb" event={"ID":"737686ff-fa6e-49d2-b9ff-e627b0e82889","Type":"ContainerStarted","Data":"417f30ad153163a17b4d23827b69adb60a035f1330a67e29518434cd6d5ee97a"}
Jul 06 21:13:06.100730 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.100708    2581 generic.go:356] "Generic (PLEG): container finished" podID="dfa5cd09-70f6-4a62-965e-647b129b69be" containerID="0092d70d3f2a8506baf2caa43e6b5e0a47b88850c04cb3786f13fa6ec1eeb9fe" exitCode=0
Jul 06 21:13:06.100844 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.100774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5lgg" event={"ID":"dfa5cd09-70f6-4a62-965e-647b129b69be","Type":"ContainerDied","Data":"0092d70d3f2a8506baf2caa43e6b5e0a47b88850c04cb3786f13fa6ec1eeb9fe"}
Jul 06 21:13:06.102417 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.102394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vnjbh" event={"ID":"c8b1bafa-cee5-4b58-8da4-753d6c0805bd","Type":"ContainerStarted","Data":"ae51bb0d28416983a43ecad8bd1e9abedd83decf592e16e81a57888502aaed76"}
Jul 06 21:13:06.104097 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.104068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc" event={"ID":"45b56ec5-de05-4b05-ad34-d6398fccca19","Type":"ContainerStarted","Data":"2506d2b8ee0d4dc37a32359cd0963c2f4e611107ca2128f97564d0092854af33"}
Jul 06 21:13:06.105595 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.105553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5thzq" event={"ID":"831d79b5-c28f-4857-8b33-0b013f995cea","Type":"ContainerStarted","Data":"cae9f8cf45aaf46c499079b52c086244ba77a4dcc261b60d38bf8ce3e3e74ef6"}
Jul 06 21:13:06.107020 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.106981    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nrrrm" event={"ID":"25fcb51b-8a18-4ba8-b525-86c727dbabf4","Type":"ContainerStarted","Data":"ff07b9cc45819eb61d0a8307bdbcbdd036c9fa2dc0ed665a62a7bec76c402b9b"}
Jul 06 21:13:06.113073 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.113033    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x2nkj" podStartSLOduration=3.366733509 podStartE2EDuration="22.113019486s" totalImagesPullingTime="18.746285977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:06.11265342 +0000 UTC m=+23.106752580" watchObservedRunningTime="2026-07-06 21:13:06.113019486 +0000 UTC m=+23.107118646"
Jul 06 21:13:06.141262 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.141183    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nrrrm" podStartSLOduration=3.408922715 podStartE2EDuration="22.14116256s" totalImagesPullingTime="18.732239845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:06.140681374 +0000 UTC m=+23.134780533" watchObservedRunningTime="2026-07-06 21:13:06.14116256 +0000 UTC m=+23.135261720"
Jul 06 21:13:06.141887 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.141849    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vnjbh" podStartSLOduration=7.637893537 podStartE2EDuration="22.14183692s" totalImagesPullingTime="14.503943383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:06.127113873 +0000 UTC m=+23.121213033" watchObservedRunningTime="2026-07-06 21:13:06.14183692 +0000 UTC m=+23.135936082"
Jul 06 21:13:06.158978 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.158921    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5thzq" podStartSLOduration=3.370466295 podStartE2EDuration="22.15890227s" totalImagesPullingTime="18.788435975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:06.158584738 +0000 UTC m=+23.152683898" watchObservedRunningTime="2026-07-06 21:13:06.15890227 +0000 UTC m=+23.153001430"
Jul 06 21:13:06.175721 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.175678    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wpfxb" podStartSLOduration=3.419038958 podStartE2EDuration="22.175663879s" totalImagesPullingTime="18.756624921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:06.175442501 +0000 UTC m=+23.169541661" watchObservedRunningTime="2026-07-06 21:13:06.175663879 +0000 UTC m=+23.169763038"
Jul 06 21:13:06.642349 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.642311    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x2nkj"
Jul 06 21:13:06.643055 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:06.643040    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x2nkj"
Jul 06 21:13:07.111421 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:07.111346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t" event={"ID":"cc05b91f-f482-4025-b0f3-d9bff11999d2","Type":"ContainerStarted","Data":"13c21db7d173d6aceb23ac4187a14ac254a8826475516834cbc705e9fda2300a"}
Jul 06 21:13:07.111421 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:07.111413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t" event={"ID":"cc05b91f-f482-4025-b0f3-d9bff11999d2","Type":"ContainerStarted","Data":"339794c5ff78c9070bb5da383753dd685cb8e7168f1548560ab019c62f90ed05"}
Jul 06 21:13:07.112333 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:07.111445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t" event={"ID":"cc05b91f-f482-4025-b0f3-d9bff11999d2","Type":"ContainerStarted","Data":"1bd42e2680849b9dc2f1d9898355dda56d05a19e1798da383f8bb096f15af59f"}
Jul 06 21:13:07.112333 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:07.111473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t" event={"ID":"cc05b91f-f482-4025-b0f3-d9bff11999d2","Type":"ContainerStarted","Data":"5ff646b8537f117e54152ff3e6d4353f3bb402aef6bac70bcfc68095a569d673"}
Jul 06 21:13:07.263932 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:07.263907    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 21:13:07.809246 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:07.809098    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T21:13:07.263924681Z","UUID":"fa157b0e-647e-47c5-bde7-da0524ed3d8e","Handler":null,"Name":"","Endpoint":""}
Jul 06 21:13:07.810983 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:07.810943    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 21:13:07.810983 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:07.810987    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 21:13:07.873460 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:07.873405    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:07.873460 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:07.873444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:07.873725 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:07.873411    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:07.873725 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:07.873524    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:13:07.873725 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:07.873619    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:13:07.873725 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:07.873704    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:13:08.115100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:08.115014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7t6dg" event={"ID":"a3c2bf1c-dda7-4db4-9a21-d1230fb66eee","Type":"ContainerStarted","Data":"94a8f940d191ac243c7c0df5e2445ab8bba2f0f39981001b82410202a7d42041"}
Jul 06 21:13:08.116574 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:08.116555    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 06 21:13:08.116678 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:08.116556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc" event={"ID":"45b56ec5-de05-4b05-ad34-d6398fccca19","Type":"ContainerStarted","Data":"d7b8bc2773da45c2756cab5919503495165c86015292e18a15744e69f5e7dca9"}
Jul 06 21:13:08.129257 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:08.129202    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7t6dg" podStartSLOduration=9.631282438 podStartE2EDuration="24.129189409s" totalImagesPullingTime="14.497906971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:08.128908434 +0000 UTC m=+25.123007615" watchObservedRunningTime="2026-07-06 21:13:08.129189409 +0000 UTC m=+25.123288569"
Jul 06 21:13:09.122062 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:09.122026    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t" event={"ID":"cc05b91f-f482-4025-b0f3-d9bff11999d2","Type":"ContainerStarted","Data":"c54c9382ff72ef8ec0d5610be6298948cd3a37b94205998401e0cc08cb110159"}
Jul 06 21:13:09.124621 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:09.124586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc" event={"ID":"45b56ec5-de05-4b05-ad34-d6398fccca19","Type":"ContainerStarted","Data":"672adc93dff05645a5be4d9fa378dd6288f8c825c0cd09de97683a14c7d78470"}
Jul 06 21:13:09.143515 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:09.143473    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf8tc" podStartSLOduration=4.228974717 podStartE2EDuration="25.143459004s" totalImagesPullingTime="20.914484287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:09.143030556 +0000 UTC m=+26.137129716" watchObservedRunningTime="2026-07-06 21:13:09.143459004 +0000 UTC m=+26.137558167"
Jul 06 21:13:09.873356 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:09.873322    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:09.873522 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:09.873325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:09.873522 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:09.873428    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:13:09.873609 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:09.873523    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:13:09.873609 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:09.873325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:09.873692 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:09.873641    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:13:11.134727 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:11.134552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t" event={"ID":"cc05b91f-f482-4025-b0f3-d9bff11999d2","Type":"ContainerStarted","Data":"ded3200c0e6e0050c58e266ac994216a40f045ddaac1263ecae6bbe22b2d80fc"}
Jul 06 21:13:11.135354 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:11.134880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:13:11.135354 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:11.134948    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:13:11.135354 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:11.134997    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:13:11.154292 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:11.154264    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:13:11.155298 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:11.155281    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:13:11.165008 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:11.164961    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t" podStartSLOduration=8.338042527 podStartE2EDuration="27.16494777s" totalImagesPullingTime="18.826905243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:11.164628919 +0000 UTC m=+28.158728094" watchObservedRunningTime="2026-07-06 21:13:11.16494777 +0000 UTC m=+28.159046929"
Jul 06 21:13:11.873414 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:11.873376    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:11.873571 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:11.873385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:11.873571 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:11.873481    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:13:11.873571 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:11.873497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:11.873703 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:11.873579    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:13:11.873703 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:11.873629    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:13:12.688097 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:12.688067    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l4qqt"]
Jul 06 21:13:12.688694 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:12.688175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:12.688694 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:12.688303    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:13:12.691368 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:12.691343    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-klq8n"]
Jul 06 21:13:12.691486 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:12.691425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:12.691540 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:12.691492    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:13:12.694477 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:12.694452    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7l279"]
Jul 06 21:13:12.694603 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:12.694540    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:12.694666 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:12.694643    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:13:13.211847 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:13.211803    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x2nkj"
Jul 06 21:13:13.212017 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:13.211935    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 06 21:13:13.212575 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:13.212553    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x2nkj"
Jul 06 21:13:13.873971 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:13.873737    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:13.874439 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:13.873810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:13.874439 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:13.874085    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:13:13.874439 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:13.874138    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:13:14.143037 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:14.142923    2581 generic.go:356] "Generic (PLEG): container finished" podID="dfa5cd09-70f6-4a62-965e-647b129b69be" containerID="6d6ccb2698bd708510e32aaf06026589d1a70d8e3ddbede0412eaf4dce3b59eb" exitCode=0
Jul 06 21:13:14.143037 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:14.143010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5lgg" event={"ID":"dfa5cd09-70f6-4a62-965e-647b129b69be","Type":"ContainerDied","Data":"6d6ccb2698bd708510e32aaf06026589d1a70d8e3ddbede0412eaf4dce3b59eb"}
Jul 06 21:13:14.872737 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:14.872709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:14.872882 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:14.872811    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:13:15.872586 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:15.872551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:15.873048 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:15.872555    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:15.873048 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:15.872707    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:13:15.873048 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:15.872767    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:13:16.150300 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:16.150265    2581 generic.go:356] "Generic (PLEG): container finished" podID="dfa5cd09-70f6-4a62-965e-647b129b69be" containerID="d9dc8e95e3aaec3cb05bad8fd663a166196da6943632304a6bdd8c96848aabc5" exitCode=0
Jul 06 21:13:16.150442 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:16.150325    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5lgg" event={"ID":"dfa5cd09-70f6-4a62-965e-647b129b69be","Type":"ContainerDied","Data":"d9dc8e95e3aaec3cb05bad8fd663a166196da6943632304a6bdd8c96848aabc5"}
Jul 06 21:13:16.872932 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:16.872907    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:16.873243 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:16.873007    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4qqt" podUID="376861d0-1d29-4779-9269-e87fdc65094a"
Jul 06 21:13:17.154481 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:17.154451    2581 generic.go:356] "Generic (PLEG): container finished" podID="dfa5cd09-70f6-4a62-965e-647b129b69be" containerID="67cf193fc6ef82bef4428fd5f75f496abbf330e879ff05ef3cda64dc9186bffd" exitCode=0
Jul 06 21:13:17.154636 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:17.154504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5lgg" event={"ID":"dfa5cd09-70f6-4a62-965e-647b129b69be","Type":"ContainerDied","Data":"67cf193fc6ef82bef4428fd5f75f496abbf330e879ff05ef3cda64dc9186bffd"}
Jul 06 21:13:17.612018 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:17.611942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nzc\" (UniqueName: \"kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc\") pod \"network-check-target-klq8n\" (UID: \"3f33c872-374e-4583-89da-6457f198a950\") " pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:17.612018 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:17.611979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:17.612179 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:17.612095    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 21:13:17.612179 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:17.612115    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 21:13:17.612179 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:17.612124    2581 projected.go:196] Error preparing data for projected volume kube-api-access-52nzc for pod openshift-network-diagnostics/network-check-target-klq8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:13:17.612179 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:17.612140    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:13:17.612179 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:17.612169    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc podName:3f33c872-374e-4583-89da-6457f198a950 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:49.612153816 +0000 UTC m=+66.606252958 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-52nzc" (UniqueName: "kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc") pod "network-check-target-klq8n" (UID: "3f33c872-374e-4583-89da-6457f198a950") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 21:13:17.612416 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:17.612182    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs podName:75a6daab-e23d-43e9-ab04-d52574901b8e nodeName:}" failed. No retries permitted until 2026-07-06 21:13:49.61217651 +0000 UTC m=+66.606275649 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs") pod "network-metrics-daemon-7l279" (UID: "75a6daab-e23d-43e9-ab04-d52574901b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 21:13:17.612416 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:17.612198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:17.612416 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:17.612325    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 21:13:17.612416 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:17.612377    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret podName:376861d0-1d29-4779-9269-e87fdc65094a nodeName:}" failed. No retries permitted until 2026-07-06 21:13:49.612359956 +0000 UTC m=+66.606459109 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret") pod "global-pull-secret-syncer-l4qqt" (UID: "376861d0-1d29-4779-9269-e87fdc65094a") : object "kube-system"/"original-pull-secret" not registered
Jul 06 21:13:17.873354 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:17.873268    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:17.873794 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:17.873276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:17.873794 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:17.873399    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klq8n" podUID="3f33c872-374e-4583-89da-6457f198a950"
Jul 06 21:13:17.873794 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:17.873513    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:13:18.725224 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.725063    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-165.ec2.internal" event="NodeReady"
Jul 06 21:13:18.725427 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.725329    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 21:13:18.757978 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.757942    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg"]
Jul 06 21:13:18.760049 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.760025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg"
Jul 06 21:13:18.761554 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.761515    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"]
Jul 06 21:13:18.762890 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.762784    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 06 21:13:18.762989 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.762917    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 06 21:13:18.763046 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.763013    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 06 21:13:18.763046 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.763016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 06 21:13:18.763186 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.763168    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rrg6q\""
Jul 06 21:13:18.764094 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.764076    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84f76685fb-ht8ct"]
Jul 06 21:13:18.764211 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.764196    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:18.765791 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.765771    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"]
Jul 06 21:13:18.765895 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.765845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:18.767247 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.767207    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 06 21:13:18.770469 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.770456    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 06 21:13:18.770540 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.770493    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7xhcl\""
Jul 06 21:13:18.770597 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.770554    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 06 21:13:18.770648 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.770612    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:18.771284 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.771263    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 06 21:13:18.774028 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.773717    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 06 21:13:18.774028 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.773758    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 06 21:13:18.774028 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.773840    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 06 21:13:18.774028 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.773921    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 06 21:13:18.774387 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.774367    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jbvdd"]
Jul 06 21:13:18.775859 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.775819    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 06 21:13:18.776090 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.775974    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9gs9l"]
Jul 06 21:13:18.776090 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.776033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jbvdd"
Jul 06 21:13:18.777609 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.777596    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:18.778400 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.778381    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 21:13:18.778507 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.778399    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 21:13:18.778507 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.778459    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4j4jp\""
Jul 06 21:13:18.778507 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.778491    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 21:13:18.779657 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.779638    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 21:13:18.779802 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.779786    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 21:13:18.779845 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.779786    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-45jgn\""
Jul 06 21:13:18.783048 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.783028    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg"]
Jul 06 21:13:18.786659 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.786572    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"]
Jul 06 21:13:18.790603 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.790583    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"]
Jul 06 21:13:18.796932 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.796910    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f76685fb-ht8ct"]
Jul 06 21:13:18.800602 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.800581    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jbvdd"]
Jul 06 21:13:18.805416 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.805394    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9gs9l"]
Jul 06 21:13:18.872730 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.872702    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:18.875197 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.875183    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 21:13:18.920447 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eda8e131-c168-47dc-9b65-591afe7cedc9-ca-trust-extracted\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:18.920447 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwt7h\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-kube-api-access-mwt7h\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:18.920646 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b18744f6-a13d-4da8-addf-9e85256d122b-hub\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:18.920646 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-bound-sa-token\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:18.920646 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b18744f6-a13d-4da8-addf-9e85256d122b-ca\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:18.920646 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eda8e131-c168-47dc-9b65-591afe7cedc9-trusted-ca\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:18.920646 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69fql\" (UniqueName: \"kubernetes.io/projected/5c08ff3c-4b2c-46c2-bd19-48069125d8d3-kube-api-access-69fql\") pod \"ingress-canary-jbvdd\" (UID: \"5c08ff3c-4b2c-46c2-bd19-48069125d8d3\") " pod="openshift-ingress-canary/ingress-canary-jbvdd"
Jul 06 21:13:18.920646 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eda8e131-c168-47dc-9b65-591afe7cedc9-installation-pull-secrets\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:18.920842 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920674    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a932825-f754-47ac-80b8-9d1e9a0b7d47-tmp-dir\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:18.920842 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20a3bd69-8cff-441d-a149-5d80353eade0-tmp\") pod \"klusterlet-addon-workmgr-5f6b9f8f8d-24lkv\" (UID: \"20a3bd69-8cff-441d-a149-5d80353eade0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:18.920842 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbgg\" (UniqueName: \"kubernetes.io/projected/df78cde0-2290-4097-9ebb-54fd8f3fbca0-kube-api-access-zxbgg\") pod \"managed-serviceaccount-addon-agent-56657dc854-wg7cg\" (UID: \"df78cde0-2290-4097-9ebb-54fd8f3fbca0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg"
Jul 06 21:13:18.920842 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b18744f6-a13d-4da8-addf-9e85256d122b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:18.920842 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c08ff3c-4b2c-46c2-bd19-48069125d8d3-cert\") pod \"ingress-canary-jbvdd\" (UID: \"5c08ff3c-4b2c-46c2-bd19-48069125d8d3\") " pod="openshift-ingress-canary/ingress-canary-jbvdd"
Jul 06 21:13:18.920842 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920801    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lfmk\" (UniqueName: \"kubernetes.io/projected/b18744f6-a13d-4da8-addf-9e85256d122b-kube-api-access-5lfmk\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:18.920842 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920826    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a932825-f754-47ac-80b8-9d1e9a0b7d47-config-volume\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:18.920842 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87fvp\" (UniqueName: \"kubernetes.io/projected/8a932825-f754-47ac-80b8-9d1e9a0b7d47-kube-api-access-87fvp\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:18.921100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920869    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b18744f6-a13d-4da8-addf-9e85256d122b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:18.921100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-certificates\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:18.921100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920943    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:18.921100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvwq\" (UniqueName: \"kubernetes.io/projected/20a3bd69-8cff-441d-a149-5d80353eade0-kube-api-access-mmvwq\") pod \"klusterlet-addon-workmgr-5f6b9f8f8d-24lkv\" (UID: \"20a3bd69-8cff-441d-a149-5d80353eade0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:18.921100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:18.921100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.920989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20a3bd69-8cff-441d-a149-5d80353eade0-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f6b9f8f8d-24lkv\" (UID: \"20a3bd69-8cff-441d-a149-5d80353eade0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:18.921100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.921010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df78cde0-2290-4097-9ebb-54fd8f3fbca0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56657dc854-wg7cg\" (UID: \"df78cde0-2290-4097-9ebb-54fd8f3fbca0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg"
Jul 06 21:13:18.921100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.921025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eda8e131-c168-47dc-9b65-591afe7cedc9-image-registry-private-configuration\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:18.921100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.921042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18744f6-a13d-4da8-addf-9e85256d122b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:18.926702 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:18.926674    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x2nkj_69673591-604c-4157-9715-fcd796eaf2a8/konnectivity-agent/0.log" limit=1024
Jul 06 21:13:19.019922 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.019827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x2nkj_69673591-604c-4157-9715-fcd796eaf2a8/konnectivity-agent/0.log" limit=1024
Jul 06 21:13:19.021818 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.021795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.021951 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.021828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvwq\" (UniqueName: \"kubernetes.io/projected/20a3bd69-8cff-441d-a149-5d80353eade0-kube-api-access-mmvwq\") pod \"klusterlet-addon-workmgr-5f6b9f8f8d-24lkv\" (UID: \"20a3bd69-8cff-441d-a149-5d80353eade0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:19.021951 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.021857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:19.021951 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.021881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20a3bd69-8cff-441d-a149-5d80353eade0-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f6b9f8f8d-24lkv\" (UID: \"20a3bd69-8cff-441d-a149-5d80353eade0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:19.022124 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:19.021970    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 21:13:19.022124 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:19.021989    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f76685fb-ht8ct: secret "image-registry-tls" not found
Jul 06 21:13:19.022124 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df78cde0-2290-4097-9ebb-54fd8f3fbca0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56657dc854-wg7cg\" (UID: \"df78cde0-2290-4097-9ebb-54fd8f3fbca0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg"
Jul 06 21:13:19.022124 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:19.022047    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls podName:eda8e131-c168-47dc-9b65-591afe7cedc9 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:19.522028919 +0000 UTC m=+36.516128075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls") pod "image-registry-84f76685fb-ht8ct" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9") : secret "image-registry-tls" not found
Jul 06 21:13:19.022124 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eda8e131-c168-47dc-9b65-591afe7cedc9-image-registry-private-configuration\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.022124 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:19.021977    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 21:13:19.022124 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18744f6-a13d-4da8-addf-9e85256d122b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.022459 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:19.022146    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls podName:8a932825-f754-47ac-80b8-9d1e9a0b7d47 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:19.522126104 +0000 UTC m=+36.516225265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls") pod "dns-default-9gs9l" (UID: "8a932825-f754-47ac-80b8-9d1e9a0b7d47") : secret "dns-default-metrics-tls" not found
Jul 06 21:13:19.022459 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eda8e131-c168-47dc-9b65-591afe7cedc9-ca-trust-extracted\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.022459 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwt7h\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-kube-api-access-mwt7h\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.022459 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b18744f6-a13d-4da8-addf-9e85256d122b-hub\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.022459 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-bound-sa-token\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.022459 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b18744f6-a13d-4da8-addf-9e85256d122b-ca\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.022459 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eda8e131-c168-47dc-9b65-591afe7cedc9-trusted-ca\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.022459 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69fql\" (UniqueName: \"kubernetes.io/projected/5c08ff3c-4b2c-46c2-bd19-48069125d8d3-kube-api-access-69fql\") pod \"ingress-canary-jbvdd\" (UID: \"5c08ff3c-4b2c-46c2-bd19-48069125d8d3\") " pod="openshift-ingress-canary/ingress-canary-jbvdd"
Jul 06 21:13:19.022459 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eda8e131-c168-47dc-9b65-591afe7cedc9-installation-pull-secrets\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.022459 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a932825-f754-47ac-80b8-9d1e9a0b7d47-tmp-dir\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:19.022459 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20a3bd69-8cff-441d-a149-5d80353eade0-tmp\") pod \"klusterlet-addon-workmgr-5f6b9f8f8d-24lkv\" (UID: \"20a3bd69-8cff-441d-a149-5d80353eade0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:19.022459 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbgg\" (UniqueName: \"kubernetes.io/projected/df78cde0-2290-4097-9ebb-54fd8f3fbca0-kube-api-access-zxbgg\") pod \"managed-serviceaccount-addon-agent-56657dc854-wg7cg\" (UID: \"df78cde0-2290-4097-9ebb-54fd8f3fbca0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg"
Jul 06 21:13:19.022973 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b18744f6-a13d-4da8-addf-9e85256d122b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.022973 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c08ff3c-4b2c-46c2-bd19-48069125d8d3-cert\") pod \"ingress-canary-jbvdd\" (UID: \"5c08ff3c-4b2c-46c2-bd19-48069125d8d3\") " pod="openshift-ingress-canary/ingress-canary-jbvdd"
Jul 06 21:13:19.022973 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lfmk\" (UniqueName: \"kubernetes.io/projected/b18744f6-a13d-4da8-addf-9e85256d122b-kube-api-access-5lfmk\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.022973 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a932825-f754-47ac-80b8-9d1e9a0b7d47-config-volume\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:19.022973 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87fvp\" (UniqueName: \"kubernetes.io/projected/8a932825-f754-47ac-80b8-9d1e9a0b7d47-kube-api-access-87fvp\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:19.022973 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b18744f6-a13d-4da8-addf-9e85256d122b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.022973 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-certificates\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.022973 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022826    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18744f6-a13d-4da8-addf-9e85256d122b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.023332 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.022978    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eda8e131-c168-47dc-9b65-591afe7cedc9-ca-trust-extracted\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.023332 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.023205    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-certificates\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.027056 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.026970    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eda8e131-c168-47dc-9b65-591afe7cedc9-trusted-ca\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.027340 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.027306    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a932825-f754-47ac-80b8-9d1e9a0b7d47-config-volume\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:19.028856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.027970    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b18744f6-a13d-4da8-addf-9e85256d122b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.028856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.028063    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b18744f6-a13d-4da8-addf-9e85256d122b-hub\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.028856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.028072    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b18744f6-a13d-4da8-addf-9e85256d122b-ca\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.028856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.028295    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a932825-f754-47ac-80b8-9d1e9a0b7d47-tmp-dir\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:19.028856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.028371    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df78cde0-2290-4097-9ebb-54fd8f3fbca0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56657dc854-wg7cg\" (UID: \"df78cde0-2290-4097-9ebb-54fd8f3fbca0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg"
Jul 06 21:13:19.028856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.028590    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eda8e131-c168-47dc-9b65-591afe7cedc9-image-registry-private-configuration\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.028856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.028599    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20a3bd69-8cff-441d-a149-5d80353eade0-tmp\") pod \"klusterlet-addon-workmgr-5f6b9f8f8d-24lkv\" (UID: \"20a3bd69-8cff-441d-a149-5d80353eade0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:19.028856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.028688    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20a3bd69-8cff-441d-a149-5d80353eade0-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f6b9f8f8d-24lkv\" (UID: \"20a3bd69-8cff-441d-a149-5d80353eade0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:19.029266 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.029162    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c08ff3c-4b2c-46c2-bd19-48069125d8d3-cert\") pod \"ingress-canary-jbvdd\" (UID: \"5c08ff3c-4b2c-46c2-bd19-48069125d8d3\") " pod="openshift-ingress-canary/ingress-canary-jbvdd"
Jul 06 21:13:19.031502 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.031471    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eda8e131-c168-47dc-9b65-591afe7cedc9-installation-pull-secrets\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.033215 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.033092    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-bound-sa-token\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.033215 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.033109    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwt7h\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-kube-api-access-mwt7h\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.034132 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.034106    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69fql\" (UniqueName: \"kubernetes.io/projected/5c08ff3c-4b2c-46c2-bd19-48069125d8d3-kube-api-access-69fql\") pod \"ingress-canary-jbvdd\" (UID: \"5c08ff3c-4b2c-46c2-bd19-48069125d8d3\") " pod="openshift-ingress-canary/ingress-canary-jbvdd"
Jul 06 21:13:19.034856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.034834    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvwq\" (UniqueName: \"kubernetes.io/projected/20a3bd69-8cff-441d-a149-5d80353eade0-kube-api-access-mmvwq\") pod \"klusterlet-addon-workmgr-5f6b9f8f8d-24lkv\" (UID: \"20a3bd69-8cff-441d-a149-5d80353eade0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:19.035189 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.035142    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b18744f6-a13d-4da8-addf-9e85256d122b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.036353 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.036331    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbgg\" (UniqueName: \"kubernetes.io/projected/df78cde0-2290-4097-9ebb-54fd8f3fbca0-kube-api-access-zxbgg\") pod \"managed-serviceaccount-addon-agent-56657dc854-wg7cg\" (UID: \"df78cde0-2290-4097-9ebb-54fd8f3fbca0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg"
Jul 06 21:13:19.037222 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.037198    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87fvp\" (UniqueName: \"kubernetes.io/projected/8a932825-f754-47ac-80b8-9d1e9a0b7d47-kube-api-access-87fvp\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:19.037811 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.037785    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lfmk\" (UniqueName: \"kubernetes.io/projected/b18744f6-a13d-4da8-addf-9e85256d122b-kube-api-access-5lfmk\") pod \"cluster-proxy-proxy-agent-659f984cd4-kxlrd\" (UID: \"b18744f6-a13d-4da8-addf-9e85256d122b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.084548 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.084512    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg"
Jul 06 21:13:19.093843 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.093820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:19.110120 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.110093    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:13:19.127340 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.127305    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jbvdd"
Jul 06 21:13:19.298863 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.298795    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg"]
Jul 06 21:13:19.302346 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.302214    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"]
Jul 06 21:13:19.306561 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:13:19.306530    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a3bd69_8cff_441d_a149_5d80353eade0.slice/crio-ac5bb8beb1f6b4998a383e65f25f3a8c473fa1674802ebf75d0156ea9fe1eceb WatchSource:0}: Error finding container ac5bb8beb1f6b4998a383e65f25f3a8c473fa1674802ebf75d0156ea9fe1eceb: Status 404 returned error can't find the container with id ac5bb8beb1f6b4998a383e65f25f3a8c473fa1674802ebf75d0156ea9fe1eceb
Jul 06 21:13:19.323490 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.323467    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jbvdd"]
Jul 06 21:13:19.327907 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:13:19.327876    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c08ff3c_4b2c_46c2_bd19_48069125d8d3.slice/crio-00ded6e9e8b87c9892c2dda951fa1a77c16c12ded739c78a700d260fb42a120a WatchSource:0}: Error finding container 00ded6e9e8b87c9892c2dda951fa1a77c16c12ded739c78a700d260fb42a120a: Status 404 returned error can't find the container with id 00ded6e9e8b87c9892c2dda951fa1a77c16c12ded739c78a700d260fb42a120a
Jul 06 21:13:19.332952 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.332899    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"]
Jul 06 21:13:19.335806 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:13:19.335772    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb18744f6_a13d_4da8_addf_9e85256d122b.slice/crio-00183f4772b6cd51cfcb6f4140a578628f5c1986e62d3964b8d5e13d35b45430 WatchSource:0}: Error finding container 00183f4772b6cd51cfcb6f4140a578628f5c1986e62d3964b8d5e13d35b45430: Status 404 returned error can't find the container with id 00183f4772b6cd51cfcb6f4140a578628f5c1986e62d3964b8d5e13d35b45430
Jul 06 21:13:19.528082 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.527995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:19.528082 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.528030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:19.528330 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:19.528156    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 21:13:19.528330 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:19.528176    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f76685fb-ht8ct: secret "image-registry-tls" not found
Jul 06 21:13:19.528330 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:19.528196    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 21:13:19.528330 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:19.528252    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls podName:eda8e131-c168-47dc-9b65-591afe7cedc9 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:20.528219685 +0000 UTC m=+37.522318845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls") pod "image-registry-84f76685fb-ht8ct" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9") : secret "image-registry-tls" not found
Jul 06 21:13:19.528330 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:19.528275    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls podName:8a932825-f754-47ac-80b8-9d1e9a0b7d47 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:20.528268656 +0000 UTC m=+37.522367794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls") pod "dns-default-9gs9l" (UID: "8a932825-f754-47ac-80b8-9d1e9a0b7d47") : secret "dns-default-metrics-tls" not found
Jul 06 21:13:19.874101 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.873258    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:19.874346 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.874328    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:19.878746 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.877837    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 21:13:19.878746 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.878050    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 21:13:19.878746 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.878275    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xnr9n\""
Jul 06 21:13:19.878746 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.878451    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qlcwx\""
Jul 06 21:13:19.878746 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:19.878615    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 21:13:20.168614 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:20.168565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv" event={"ID":"20a3bd69-8cff-441d-a149-5d80353eade0","Type":"ContainerStarted","Data":"ac5bb8beb1f6b4998a383e65f25f3a8c473fa1674802ebf75d0156ea9fe1eceb"}
Jul 06 21:13:20.173993 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:20.173959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg" event={"ID":"df78cde0-2290-4097-9ebb-54fd8f3fbca0","Type":"ContainerStarted","Data":"64511d977df2c53ea3d1232eaeb28a9a0e544874cd0e908cb7797352c9ac28e8"}
Jul 06 21:13:20.178722 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:20.178589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jbvdd" event={"ID":"5c08ff3c-4b2c-46c2-bd19-48069125d8d3","Type":"ContainerStarted","Data":"00ded6e9e8b87c9892c2dda951fa1a77c16c12ded739c78a700d260fb42a120a"}
Jul 06 21:13:20.184791 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:20.184717    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd" event={"ID":"b18744f6-a13d-4da8-addf-9e85256d122b","Type":"ContainerStarted","Data":"00183f4772b6cd51cfcb6f4140a578628f5c1986e62d3964b8d5e13d35b45430"}
Jul 06 21:13:20.539345 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:20.539262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:20.539345 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:20.539316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:20.539613 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:20.539449    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 21:13:20.539613 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:20.539507    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls podName:8a932825-f754-47ac-80b8-9d1e9a0b7d47 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:22.539489554 +0000 UTC m=+39.533588707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls") pod "dns-default-9gs9l" (UID: "8a932825-f754-47ac-80b8-9d1e9a0b7d47") : secret "dns-default-metrics-tls" not found
Jul 06 21:13:20.539963 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:20.539945    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 21:13:20.540041 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:20.539966    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f76685fb-ht8ct: secret "image-registry-tls" not found
Jul 06 21:13:20.540041 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:20.540011    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls podName:eda8e131-c168-47dc-9b65-591afe7cedc9 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:22.539996569 +0000 UTC m=+39.534095722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls") pod "image-registry-84f76685fb-ht8ct" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9") : secret "image-registry-tls" not found
Jul 06 21:13:22.559687 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:22.558774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:22.559687 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:22.559048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:22.559687 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:22.558919    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 21:13:22.559687 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:22.559219    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f76685fb-ht8ct: secret "image-registry-tls" not found
Jul 06 21:13:22.559687 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:22.559291    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls podName:eda8e131-c168-47dc-9b65-591afe7cedc9 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:26.559274423 +0000 UTC m=+43.553373581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls") pod "image-registry-84f76685fb-ht8ct" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9") : secret "image-registry-tls" not found
Jul 06 21:13:22.559687 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:22.559203    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 21:13:22.559687 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:22.559655    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls podName:8a932825-f754-47ac-80b8-9d1e9a0b7d47 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:26.559635912 +0000 UTC m=+43.553735063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls") pod "dns-default-9gs9l" (UID: "8a932825-f754-47ac-80b8-9d1e9a0b7d47") : secret "dns-default-metrics-tls" not found
Jul 06 21:13:26.587958 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:26.587918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:26.588402 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:26.587975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:26.588402 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:26.588079    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 21:13:26.588402 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:26.588105    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f76685fb-ht8ct: secret "image-registry-tls" not found
Jul 06 21:13:26.588402 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:26.588105    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 21:13:26.588402 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:26.588172    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls podName:8a932825-f754-47ac-80b8-9d1e9a0b7d47 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:34.588150229 +0000 UTC m=+51.582249367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls") pod "dns-default-9gs9l" (UID: "8a932825-f754-47ac-80b8-9d1e9a0b7d47") : secret "dns-default-metrics-tls" not found
Jul 06 21:13:26.588402 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:26.588189    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls podName:eda8e131-c168-47dc-9b65-591afe7cedc9 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:34.588183236 +0000 UTC m=+51.582282373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls") pod "image-registry-84f76685fb-ht8ct" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9") : secret "image-registry-tls" not found
Jul 06 21:13:29.216099 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:29.216014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd" event={"ID":"b18744f6-a13d-4da8-addf-9e85256d122b","Type":"ContainerStarted","Data":"730e9ce84b811aee47b63738bbb10a65b39741e35b8b1b9ac5f27518f7d35c85"}
Jul 06 21:13:29.220528 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:29.220488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5lgg" event={"ID":"dfa5cd09-70f6-4a62-965e-647b129b69be","Type":"ContainerStarted","Data":"e61db59288631bb10e58b3c1535aa9c36811dc5d8001cddd8c9549e92aa74082"}
Jul 06 21:13:29.221911 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:29.221884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv" event={"ID":"20a3bd69-8cff-441d-a149-5d80353eade0","Type":"ContainerStarted","Data":"5be53ae8e39d334afdcfa92eb550590eb302db86b817c0e10e9a9eb5be7179fe"}
Jul 06 21:13:29.222222 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:29.222171    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:29.223600 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:29.223585    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:13:29.223709 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:29.223617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg" event={"ID":"df78cde0-2290-4097-9ebb-54fd8f3fbca0","Type":"ContainerStarted","Data":"46958524c267d38a7505bc6dd2fd44f13ef27b4d86ef697976eaee7bba5fd065"}
Jul 06 21:13:29.225880 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:29.225784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jbvdd" event={"ID":"5c08ff3c-4b2c-46c2-bd19-48069125d8d3","Type":"ContainerStarted","Data":"326d22e01199d363d574d99152e6af7472a238d95bc8b41c115b746f721fc0ca"}
Jul 06 21:13:29.257858 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:29.257791    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg" podStartSLOduration=22.605341134 podStartE2EDuration="32.257775957s" totalImagesPullingTime="9.652434823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:29.257535729 +0000 UTC m=+46.251634889" watchObservedRunningTime="2026-07-06 21:13:29.257775957 +0000 UTC m=+46.251875118"
Jul 06 21:13:29.294495 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:29.294450    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv" podStartSLOduration=22.627073371 podStartE2EDuration="32.294433322s" totalImagesPullingTime="9.667359951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:29.274408161 +0000 UTC m=+46.268507323" watchObservedRunningTime="2026-07-06 21:13:29.294433322 +0000 UTC m=+46.288532481"
Jul 06 21:13:29.295349 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:29.295309    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jbvdd" podStartSLOduration=1.663637306 podStartE2EDuration="11.295298165s" totalImagesPullingTime="9.631660859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:29.294331563 +0000 UTC m=+46.288430724" watchObservedRunningTime="2026-07-06 21:13:29.295298165 +0000 UTC m=+46.289397319"
Jul 06 21:13:30.230620 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:30.230580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd" event={"ID":"b18744f6-a13d-4da8-addf-9e85256d122b","Type":"ContainerStarted","Data":"600213b6dc88eedb28b52f59c441a741001d1eff482079b128f6682f11086c2f"}
Jul 06 21:13:30.230620 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:30.230629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd" event={"ID":"b18744f6-a13d-4da8-addf-9e85256d122b","Type":"ContainerStarted","Data":"037391ccd4166d0cc862def0d6aa68d63aa4d42fe295c8102943b556452f9352"}
Jul 06 21:13:30.233056 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:30.233027    2581 generic.go:356] "Generic (PLEG): container finished" podID="dfa5cd09-70f6-4a62-965e-647b129b69be" containerID="e61db59288631bb10e58b3c1535aa9c36811dc5d8001cddd8c9549e92aa74082" exitCode=0
Jul 06 21:13:30.233169 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:30.233116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5lgg" event={"ID":"dfa5cd09-70f6-4a62-965e-647b129b69be","Type":"ContainerDied","Data":"e61db59288631bb10e58b3c1535aa9c36811dc5d8001cddd8c9549e92aa74082"}
Jul 06 21:13:30.251981 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:30.251902    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd" podStartSLOduration=23.6322422 podStartE2EDuration="33.251889525s" totalImagesPullingTime="9.619647325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:30.251754878 +0000 UTC m=+47.245854038" watchObservedRunningTime="2026-07-06 21:13:30.251889525 +0000 UTC m=+47.245988684"
Jul 06 21:13:31.238985 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:31.238949    2581 generic.go:356] "Generic (PLEG): container finished" podID="dfa5cd09-70f6-4a62-965e-647b129b69be" containerID="2c270b83e0c76809067daf48fda674fafab75b0b553669ccb9047ba7e53dba29" exitCode=0
Jul 06 21:13:31.239455 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:31.239021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5lgg" event={"ID":"dfa5cd09-70f6-4a62-965e-647b129b69be","Type":"ContainerDied","Data":"2c270b83e0c76809067daf48fda674fafab75b0b553669ccb9047ba7e53dba29"}
Jul 06 21:13:32.245916 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:32.245886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5lgg" event={"ID":"dfa5cd09-70f6-4a62-965e-647b129b69be","Type":"ContainerStarted","Data":"bec0fda2c8d3b2a84c6472a62d126d248bd3c3452e5e8572fbf3daa493adb6dd"}
Jul 06 21:13:32.272269 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:32.272191    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s5lgg" podStartSLOduration=8.27442057 podStartE2EDuration="48.272171791s" totalImagesPullingTime="39.997751221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:32.270149163 +0000 UTC m=+49.264248348" watchObservedRunningTime="2026-07-06 21:13:32.272171791 +0000 UTC m=+49.266270953"
Jul 06 21:13:34.648102 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:34.648063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:34.648102 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:34.648104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:34.648573 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:34.648204    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 21:13:34.648573 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:34.648224    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f76685fb-ht8ct: secret "image-registry-tls" not found
Jul 06 21:13:34.648573 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:34.648214    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 21:13:34.648573 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:34.648298    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls podName:eda8e131-c168-47dc-9b65-591afe7cedc9 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:50.648280228 +0000 UTC m=+67.642379382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls") pod "image-registry-84f76685fb-ht8ct" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9") : secret "image-registry-tls" not found
Jul 06 21:13:34.648573 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:34.648339    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls podName:8a932825-f754-47ac-80b8-9d1e9a0b7d47 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:50.648326853 +0000 UTC m=+67.642425995 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls") pod "dns-default-9gs9l" (UID: "8a932825-f754-47ac-80b8-9d1e9a0b7d47") : secret "dns-default-metrics-tls" not found
Jul 06 21:13:43.152390 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:43.152360    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6p6t"
Jul 06 21:13:49.661555 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.661513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:49.661965 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.661594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nzc\" (UniqueName: \"kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc\") pod \"network-check-target-klq8n\" (UID: \"3f33c872-374e-4583-89da-6457f198a950\") " pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:49.661965 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.661613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:13:49.664401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.664378    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 21:13:49.664523 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.664377    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 21:13:49.664523 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.664425    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 21:13:49.672523 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:49.672504    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 21:13:49.672576 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:49.672563    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs podName:75a6daab-e23d-43e9-ab04-d52574901b8e nodeName:}" failed. No retries permitted until 2026-07-06 21:14:53.672546795 +0000 UTC m=+130.666645933 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs") pod "network-metrics-daemon-7l279" (UID: "75a6daab-e23d-43e9-ab04-d52574901b8e") : secret "metrics-daemon-secret" not found
Jul 06 21:13:49.674836 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.674822    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 21:13:49.675403 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.675384    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376861d0-1d29-4779-9269-e87fdc65094a-original-pull-secret\") pod \"global-pull-secret-syncer-l4qqt\" (UID: \"376861d0-1d29-4779-9269-e87fdc65094a\") " pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:49.684666 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.684641    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52nzc\" (UniqueName: \"kubernetes.io/projected/3f33c872-374e-4583-89da-6457f198a950-kube-api-access-52nzc\") pod \"network-check-target-klq8n\" (UID: \"3f33c872-374e-4583-89da-6457f198a950\") " pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:49.780306 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.780270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4qqt"
Jul 06 21:13:49.895125 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.895097    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xnr9n\""
Jul 06 21:13:49.898270 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.898243    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l4qqt"]
Jul 06 21:13:49.901884 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:13:49.901860    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376861d0_1d29_4779_9269_e87fdc65094a.slice/crio-b4677a804a85fa2b403f7bacd0d2b2ab40d0712ba42ce9989eccce4e224805fa WatchSource:0}: Error finding container b4677a804a85fa2b403f7bacd0d2b2ab40d0712ba42ce9989eccce4e224805fa: Status 404 returned error can't find the container with id b4677a804a85fa2b403f7bacd0d2b2ab40d0712ba42ce9989eccce4e224805fa
Jul 06 21:13:49.902879 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:49.902863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:50.021187 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:50.021156    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-klq8n"]
Jul 06 21:13:50.024014 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:13:50.023991    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f33c872_374e_4583_89da_6457f198a950.slice/crio-83b3bb7c4a590c43ba726e5fbffe8c40630ee8430e789054b3b28563fddcb684 WatchSource:0}: Error finding container 83b3bb7c4a590c43ba726e5fbffe8c40630ee8430e789054b3b28563fddcb684: Status 404 returned error can't find the container with id 83b3bb7c4a590c43ba726e5fbffe8c40630ee8430e789054b3b28563fddcb684
Jul 06 21:13:50.300864 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:50.300767    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l4qqt" event={"ID":"376861d0-1d29-4779-9269-e87fdc65094a","Type":"ContainerStarted","Data":"b4677a804a85fa2b403f7bacd0d2b2ab40d0712ba42ce9989eccce4e224805fa"}
Jul 06 21:13:50.302745 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:50.302712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-klq8n" event={"ID":"3f33c872-374e-4583-89da-6457f198a950","Type":"ContainerStarted","Data":"83b3bb7c4a590c43ba726e5fbffe8c40630ee8430e789054b3b28563fddcb684"}
Jul 06 21:13:50.670980 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:50.670928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:13:50.671462 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:50.670985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:13:50.671462 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:50.671117    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 21:13:50.671462 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:50.671120    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 21:13:50.671462 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:50.671178    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls podName:8a932825-f754-47ac-80b8-9d1e9a0b7d47 nodeName:}" failed. No retries permitted until 2026-07-06 21:14:22.671159422 +0000 UTC m=+99.665258576 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls") pod "dns-default-9gs9l" (UID: "8a932825-f754-47ac-80b8-9d1e9a0b7d47") : secret "dns-default-metrics-tls" not found
Jul 06 21:13:50.671462 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:50.671187    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f76685fb-ht8ct: secret "image-registry-tls" not found
Jul 06 21:13:50.671462 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:13:50.671278    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls podName:eda8e131-c168-47dc-9b65-591afe7cedc9 nodeName:}" failed. No retries permitted until 2026-07-06 21:14:22.671259192 +0000 UTC m=+99.665358346 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls") pod "image-registry-84f76685fb-ht8ct" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9") : secret "image-registry-tls" not found
Jul 06 21:13:55.320116 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:55.320076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l4qqt" event={"ID":"376861d0-1d29-4779-9269-e87fdc65094a","Type":"ContainerStarted","Data":"6f29ddebecaa4438129f36ffe1f2262ea1b007a4624e102bc25ea1cbdd171753"}
Jul 06 21:13:55.321450 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:55.321420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-klq8n" event={"ID":"3f33c872-374e-4583-89da-6457f198a950","Type":"ContainerStarted","Data":"7fdcc237a389c50992930310a279d4ad9f6e20f1690d495fa69a062dda380d34"}
Jul 06 21:13:55.321583 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:55.321565    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:13:55.335242 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:55.335184    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l4qqt" podStartSLOduration=66.486494308 podStartE2EDuration="1m11.335168961s" totalImagesPullingTime="4.848674653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:55.334771752 +0000 UTC m=+72.328870913" watchObservedRunningTime="2026-07-06 21:13:55.335168961 +0000 UTC m=+72.329268120"
Jul 06 21:13:55.348615 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:13:55.348563    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-klq8n" podStartSLOduration=66.621809272 podStartE2EDuration="1m11.348536136s" totalImagesPullingTime="4.726726864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:55.348519972 +0000 UTC m=+72.342619145" watchObservedRunningTime="2026-07-06 21:13:55.348536136 +0000 UTC m=+72.342635296"
Jul 06 21:14:22.699394 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:14:22.699355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:14:22.699822 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:14:22.699405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:14:22.699822 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:14:22.699508    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 21:14:22.699822 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:14:22.699521    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 21:14:22.699822 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:14:22.699526    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f76685fb-ht8ct: secret "image-registry-tls" not found
Jul 06 21:14:22.699822 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:14:22.699585    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls podName:8a932825-f754-47ac-80b8-9d1e9a0b7d47 nodeName:}" failed. No retries permitted until 2026-07-06 21:15:26.699568579 +0000 UTC m=+163.693667731 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls") pod "dns-default-9gs9l" (UID: "8a932825-f754-47ac-80b8-9d1e9a0b7d47") : secret "dns-default-metrics-tls" not found
Jul 06 21:14:22.699822 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:14:22.699598    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls podName:eda8e131-c168-47dc-9b65-591afe7cedc9 nodeName:}" failed. No retries permitted until 2026-07-06 21:15:26.699592918 +0000 UTC m=+163.693692056 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls") pod "image-registry-84f76685fb-ht8ct" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9") : secret "image-registry-tls" not found
Jul 06 21:14:26.326582 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:14:26.326556    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-klq8n"
Jul 06 21:14:53.718812 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:14:53.718771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:14:53.719347 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:14:53.718930    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 21:14:53.719347 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:14:53.719006    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs podName:75a6daab-e23d-43e9-ab04-d52574901b8e nodeName:}" failed. No retries permitted until 2026-07-06 21:16:55.718989856 +0000 UTC m=+252.713088997 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs") pod "network-metrics-daemon-7l279" (UID: "75a6daab-e23d-43e9-ab04-d52574901b8e") : secret "metrics-daemon-secret" not found
Jul 06 21:15:21.801739 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:15:21.801688    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct" podUID="eda8e131-c168-47dc-9b65-591afe7cedc9"
Jul 06 21:15:21.833015 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:15:21.832972    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9gs9l" podUID="8a932825-f754-47ac-80b8-9d1e9a0b7d47"
Jul 06 21:15:22.568504 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:22.568429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:15:22.568641 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:22.568429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:15:22.904617 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:15:22.904583    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7l279" podUID="75a6daab-e23d-43e9-ab04-d52574901b8e"
Jul 06 21:15:26.749035 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:26.748985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:15:26.749035 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:26.749036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:15:26.749500 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:15:26.749132    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 21:15:26.749500 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:15:26.749141    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 21:15:26.749500 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:15:26.749159    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f76685fb-ht8ct: secret "image-registry-tls" not found
Jul 06 21:15:26.749500 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:15:26.749192    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls podName:8a932825-f754-47ac-80b8-9d1e9a0b7d47 nodeName:}" failed. No retries permitted until 2026-07-06 21:17:28.74917784 +0000 UTC m=+285.743276978 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls") pod "dns-default-9gs9l" (UID: "8a932825-f754-47ac-80b8-9d1e9a0b7d47") : secret "dns-default-metrics-tls" not found
Jul 06 21:15:26.749500 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:15:26.749206    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls podName:eda8e131-c168-47dc-9b65-591afe7cedc9 nodeName:}" failed. No retries permitted until 2026-07-06 21:17:28.749199962 +0000 UTC m=+285.743299100 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls") pod "image-registry-84f76685fb-ht8ct" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9") : secret "image-registry-tls" not found
Jul 06 21:15:29.223359 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:29.223302    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv" podUID="20a3bd69-8cff-441d-a149-5d80353eade0" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.7:8000/readyz\": dial tcp 10.132.0.7:8000: connect: connection refused"
Jul 06 21:15:29.589979 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:29.589898    2581 generic.go:356] "Generic (PLEG): container finished" podID="20a3bd69-8cff-441d-a149-5d80353eade0" containerID="5be53ae8e39d334afdcfa92eb550590eb302db86b817c0e10e9a9eb5be7179fe" exitCode=1
Jul 06 21:15:29.590133 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:29.589973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv" event={"ID":"20a3bd69-8cff-441d-a149-5d80353eade0","Type":"ContainerDied","Data":"5be53ae8e39d334afdcfa92eb550590eb302db86b817c0e10e9a9eb5be7179fe"}
Jul 06 21:15:29.590377 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:29.590364    2581 scope.go:117] "RemoveContainer" containerID="5be53ae8e39d334afdcfa92eb550590eb302db86b817c0e10e9a9eb5be7179fe"
Jul 06 21:15:29.591388 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:29.591370    2581 generic.go:356] "Generic (PLEG): container finished" podID="df78cde0-2290-4097-9ebb-54fd8f3fbca0" containerID="46958524c267d38a7505bc6dd2fd44f13ef27b4d86ef697976eaee7bba5fd065" exitCode=255
Jul 06 21:15:29.591487 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:29.591426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg" event={"ID":"df78cde0-2290-4097-9ebb-54fd8f3fbca0","Type":"ContainerDied","Data":"46958524c267d38a7505bc6dd2fd44f13ef27b4d86ef697976eaee7bba5fd065"}
Jul 06 21:15:29.591741 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:29.591726    2581 scope.go:117] "RemoveContainer" containerID="46958524c267d38a7505bc6dd2fd44f13ef27b4d86ef697976eaee7bba5fd065"
Jul 06 21:15:30.596080 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:30.596041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv" event={"ID":"20a3bd69-8cff-441d-a149-5d80353eade0","Type":"ContainerStarted","Data":"4ef922f5938dd437d451df00de785b54315d0f7922516e7fbc7369c1e79e813f"}
Jul 06 21:15:30.596590 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:30.596358    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:15:30.596940 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:30.596925    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:15:30.597747 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:30.597726    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg" event={"ID":"df78cde0-2290-4097-9ebb-54fd8f3fbca0","Type":"ContainerStarted","Data":"2f620de1b3210035661d0562134a7afe3dc9239ca9500d7cd77954b734141012"}
Jul 06 21:15:33.874437 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:15:33.874399    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:16:35.739191 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.739156    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rpsbv"]
Jul 06 21:16:35.741111 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.741089    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.751301 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.751264    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 21:16:35.751301 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.751287    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 21:16:35.751513 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.751316    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 21:16:35.751513 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.751365    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 21:16:35.752343 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.752328    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c69zl\""
Jul 06 21:16:35.779089 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.779059    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rpsbv"]
Jul 06 21:16:35.836119 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.836082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0339a918-5080-42bb-acf8-c610c7db7ca6-data-volume\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.836349 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.836130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwwtk\" (UniqueName: \"kubernetes.io/projected/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-api-access-pwwtk\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.836349 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.836242    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.836349 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.836289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0339a918-5080-42bb-acf8-c610c7db7ca6-crio-socket\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.836349 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.836330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.936982 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.936943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.936982 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.936989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0339a918-5080-42bb-acf8-c610c7db7ca6-crio-socket\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.937287 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.937005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.937287 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:35.937055    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 21:16:35.937287 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:35.937092    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 21:16:35.937287 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.937098    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0339a918-5080-42bb-acf8-c610c7db7ca6-crio-socket\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.937287 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:35.937122    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm podName:0339a918-5080-42bb-acf8-c610c7db7ca6 nodeName:}" failed. No retries permitted until 2026-07-06 21:16:36.437103557 +0000 UTC m=+233.431202722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpsbv" (UID: "0339a918-5080-42bb-acf8-c610c7db7ca6") : configmap "kube-rbac-proxy" not found
Jul 06 21:16:35.937287 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.937149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0339a918-5080-42bb-acf8-c610c7db7ca6-data-volume\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.937287 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:35.937213    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls podName:0339a918-5080-42bb-acf8-c610c7db7ca6 nodeName:}" failed. No retries permitted until 2026-07-06 21:16:36.437191834 +0000 UTC m=+233.431290983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpsbv" (UID: "0339a918-5080-42bb-acf8-c610c7db7ca6") : secret "insights-runtime-extractor-tls" not found
Jul 06 21:16:35.937551 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.937298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwwtk\" (UniqueName: \"kubernetes.io/projected/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-api-access-pwwtk\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.937551 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.937438    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0339a918-5080-42bb-acf8-c610c7db7ca6-data-volume\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:35.954857 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:35.954824    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwwtk\" (UniqueName: \"kubernetes.io/projected/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-api-access-pwwtk\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:36.440962 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:36.440922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:36.440962 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:36.440968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:36.441260 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:36.441052    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 21:16:36.441260 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:36.441067    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 21:16:36.441260 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:36.441141    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm podName:0339a918-5080-42bb-acf8-c610c7db7ca6 nodeName:}" failed. No retries permitted until 2026-07-06 21:16:37.441121661 +0000 UTC m=+234.435220804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpsbv" (UID: "0339a918-5080-42bb-acf8-c610c7db7ca6") : configmap "kube-rbac-proxy" not found
Jul 06 21:16:36.441260 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:36.441188    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls podName:0339a918-5080-42bb-acf8-c610c7db7ca6 nodeName:}" failed. No retries permitted until 2026-07-06 21:16:37.441173049 +0000 UTC m=+234.435272187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpsbv" (UID: "0339a918-5080-42bb-acf8-c610c7db7ca6") : secret "insights-runtime-extractor-tls" not found
Jul 06 21:16:37.449324 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:37.449283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:37.449324 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:37.449326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:37.449840 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:37.449403    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 21:16:37.449840 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:37.449420    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 21:16:37.449840 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:37.449468    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm podName:0339a918-5080-42bb-acf8-c610c7db7ca6 nodeName:}" failed. No retries permitted until 2026-07-06 21:16:39.449452139 +0000 UTC m=+236.443551277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpsbv" (UID: "0339a918-5080-42bb-acf8-c610c7db7ca6") : configmap "kube-rbac-proxy" not found
Jul 06 21:16:37.449840 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:37.449482    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls podName:0339a918-5080-42bb-acf8-c610c7db7ca6 nodeName:}" failed. No retries permitted until 2026-07-06 21:16:39.449476715 +0000 UTC m=+236.443575853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpsbv" (UID: "0339a918-5080-42bb-acf8-c610c7db7ca6") : secret "insights-runtime-extractor-tls" not found
Jul 06 21:16:37.944756 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:37.944728    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vnjbh_c8b1bafa-cee5-4b58-8da4-753d6c0805bd/dns-node-resolver/0.log"
Jul 06 21:16:38.947607 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:38.947579    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nrrrm_25fcb51b-8a18-4ba8-b525-86c727dbabf4/node-ca/0.log"
Jul 06 21:16:39.465624 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:39.465593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:39.465624 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:39.465629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:39.465814 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:39.465721    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 21:16:39.465814 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:39.465725    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 21:16:39.465814 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:39.465780    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm podName:0339a918-5080-42bb-acf8-c610c7db7ca6 nodeName:}" failed. No retries permitted until 2026-07-06 21:16:43.465765922 +0000 UTC m=+240.459865063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpsbv" (UID: "0339a918-5080-42bb-acf8-c610c7db7ca6") : configmap "kube-rbac-proxy" not found
Jul 06 21:16:39.465814 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:39.465811    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls podName:0339a918-5080-42bb-acf8-c610c7db7ca6 nodeName:}" failed. No retries permitted until 2026-07-06 21:16:43.465797245 +0000 UTC m=+240.459896384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpsbv" (UID: "0339a918-5080-42bb-acf8-c610c7db7ca6") : secret "insights-runtime-extractor-tls" not found
Jul 06 21:16:39.745393 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:39.745319    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jbvdd_5c08ff3c-4b2c-46c2-bd19-48069125d8d3/serve-healthcheck-canary/0.log"
Jul 06 21:16:43.496360 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:43.496320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:43.496360 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:43.496366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:43.496852 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:43.496440    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 21:16:43.496852 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:43.496492    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 21:16:43.496852 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:43.496512    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm podName:0339a918-5080-42bb-acf8-c610c7db7ca6 nodeName:}" failed. No retries permitted until 2026-07-06 21:16:51.496495638 +0000 UTC m=+248.490594782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpsbv" (UID: "0339a918-5080-42bb-acf8-c610c7db7ca6") : configmap "kube-rbac-proxy" not found
Jul 06 21:16:43.496852 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:16:43.496534    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls podName:0339a918-5080-42bb-acf8-c610c7db7ca6 nodeName:}" failed. No retries permitted until 2026-07-06 21:16:51.496522943 +0000 UTC m=+248.490622084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpsbv" (UID: "0339a918-5080-42bb-acf8-c610c7db7ca6") : secret "insights-runtime-extractor-tls" not found
Jul 06 21:16:51.556702 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:51.556665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:51.556702 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:51.556706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:51.557208 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:51.557173    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0339a918-5080-42bb-acf8-c610c7db7ca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:51.558916 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:51.558900    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0339a918-5080-42bb-acf8-c610c7db7ca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpsbv\" (UID: \"0339a918-5080-42bb-acf8-c610c7db7ca6\") " pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:51.652427 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:51.652395    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c69zl\""
Jul 06 21:16:51.660617 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:51.660593    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rpsbv"
Jul 06 21:16:51.783175 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:51.783142    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rpsbv"]
Jul 06 21:16:51.786297 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:16:51.786264    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0339a918_5080_42bb_acf8_c610c7db7ca6.slice/crio-9f5bfcbda145fb18457f7f757c0bdad952fa2dc25dfb86d3863785090d83cf97 WatchSource:0}: Error finding container 9f5bfcbda145fb18457f7f757c0bdad952fa2dc25dfb86d3863785090d83cf97: Status 404 returned error can't find the container with id 9f5bfcbda145fb18457f7f757c0bdad952fa2dc25dfb86d3863785090d83cf97
Jul 06 21:16:51.833554 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:51.833472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rpsbv" event={"ID":"0339a918-5080-42bb-acf8-c610c7db7ca6","Type":"ContainerStarted","Data":"9f5bfcbda145fb18457f7f757c0bdad952fa2dc25dfb86d3863785090d83cf97"}
Jul 06 21:16:55.790195 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:55.790151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:16:55.792503 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:55.792477    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a6daab-e23d-43e9-ab04-d52574901b8e-metrics-certs\") pod \"network-metrics-daemon-7l279\" (UID: \"75a6daab-e23d-43e9-ab04-d52574901b8e\") " pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:16:55.847450 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:55.847415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rpsbv" event={"ID":"0339a918-5080-42bb-acf8-c610c7db7ca6","Type":"ContainerStarted","Data":"a53ad778ee5512298726e242c37428261c6b9a137c6c9f2135ef8f1a8baeccc6"}
Jul 06 21:16:56.077879 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:56.077798    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qlcwx\""
Jul 06 21:16:56.085666 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:56.085646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l279"
Jul 06 21:16:56.210761 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:56.210731    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7l279"]
Jul 06 21:16:56.213563 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:16:56.213539    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a6daab_e23d_43e9_ab04_d52574901b8e.slice/crio-4a0a82e2835a4cfbe8bef0bee48f1790940317ca70547d2e6a8bcdf8aa42fcc4 WatchSource:0}: Error finding container 4a0a82e2835a4cfbe8bef0bee48f1790940317ca70547d2e6a8bcdf8aa42fcc4: Status 404 returned error can't find the container with id 4a0a82e2835a4cfbe8bef0bee48f1790940317ca70547d2e6a8bcdf8aa42fcc4
Jul 06 21:16:56.853668 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:56.853626    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rpsbv" event={"ID":"0339a918-5080-42bb-acf8-c610c7db7ca6","Type":"ContainerStarted","Data":"e1d1b8d295d4672d94c0c0c0622d8c9713f34f0685f1a1c5c5c8cdf6e3a35b59"}
Jul 06 21:16:56.854882 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:56.854842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7l279" event={"ID":"75a6daab-e23d-43e9-ab04-d52574901b8e","Type":"ContainerStarted","Data":"4a0a82e2835a4cfbe8bef0bee48f1790940317ca70547d2e6a8bcdf8aa42fcc4"}
Jul 06 21:16:58.863042 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:58.863002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rpsbv" event={"ID":"0339a918-5080-42bb-acf8-c610c7db7ca6","Type":"ContainerStarted","Data":"6b3a2ff88d0d012ffecdfa9f331fc8aa03dcd2f7c61d4f47ce982571f43e84d0"}
Jul 06 21:16:58.864780 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:58.864747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7l279" event={"ID":"75a6daab-e23d-43e9-ab04-d52574901b8e","Type":"ContainerStarted","Data":"78c50f9baa88caf20114af82b75823b51efc82c00e67ab45407054b7f3ddf10a"}
Jul 06 21:16:58.864923 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:58.864792    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7l279" event={"ID":"75a6daab-e23d-43e9-ab04-d52574901b8e","Type":"ContainerStarted","Data":"cbf43e362029f0ac3f870248493fbcbd6e3711f6c724f73c2d3d11c225899664"}
Jul 06 21:16:58.882995 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:58.882944    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rpsbv" podStartSLOduration=17.379310405 podStartE2EDuration="23.882930786s" totalImagesPullingTime="6.503620381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:16:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:16:58.881782195 +0000 UTC m=+255.875881357" watchObservedRunningTime="2026-07-06 21:16:58.882930786 +0000 UTC m=+255.877029946"
Jul 06 21:16:58.900348 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:16:58.900303    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7l279" podStartSLOduration=252.730719895 podStartE2EDuration="4m14.900279215s" totalImagesPullingTime="2.16955932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:16:58.899263883 +0000 UTC m=+255.893363042" watchObservedRunningTime="2026-07-06 21:16:58.900279215 +0000 UTC m=+255.894378378"
Jul 06 21:17:04.540160 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.540121    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xpqt9"]
Jul 06 21:17:04.542344 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.542321    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.544939 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.544914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 21:17:04.545297 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.545283    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 21:17:04.545388 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.545345    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 21:17:04.545388 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.545356    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-brw6k\""
Jul 06 21:17:04.545388 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.545370    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 21:17:04.545883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.545864    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 21:17:04.546201 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.546188    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 21:17:04.661312 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.661273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-metrics-client-ca\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.661312 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.661317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-textfile\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.661530 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.661335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nqsq\" (UniqueName: \"kubernetes.io/projected/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-kube-api-access-2nqsq\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.661530 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.661414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-root\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.661530 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.661439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-wtmp\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.661530 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.661456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-sys\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.661530 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.661476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.661530 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.661504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.661727 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.661561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-tls\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.762861 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.762823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-root\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.762861 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.762860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-wtmp\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763071 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.762877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-sys\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763071 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.762896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763071 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.762917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763071 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.762946    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-root\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763071 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.762968    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-sys\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763071 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.762963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-tls\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763071 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.763011    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-wtmp\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763071 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.763031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-metrics-client-ca\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763452 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.763177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-textfile\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763452 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.763224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nqsq\" (UniqueName: \"kubernetes.io/projected/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-kube-api-access-2nqsq\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763550 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.763472    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-textfile\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763618 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.763597    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.763670 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.763617    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-metrics-client-ca\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.765364 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.765344    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.765525 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.765506    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-node-exporter-tls\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.770880 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.770860    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nqsq\" (UniqueName: \"kubernetes.io/projected/7b673fdb-9f97-4c99-ac99-aef2386ef9ed-kube-api-access-2nqsq\") pod \"node-exporter-xpqt9\" (UID: \"7b673fdb-9f97-4c99-ac99-aef2386ef9ed\") " pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.851259 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.851169    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xpqt9"
Jul 06 21:17:04.860275 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:17:04.860251    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b673fdb_9f97_4c99_ac99_aef2386ef9ed.slice/crio-ccfd67dbd462039cc8b3d1136ab8c277bcd8f42fc7db35d57267c478e921b2a1 WatchSource:0}: Error finding container ccfd67dbd462039cc8b3d1136ab8c277bcd8f42fc7db35d57267c478e921b2a1: Status 404 returned error can't find the container with id ccfd67dbd462039cc8b3d1136ab8c277bcd8f42fc7db35d57267c478e921b2a1
Jul 06 21:17:04.884966 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:04.884938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpqt9" event={"ID":"7b673fdb-9f97-4c99-ac99-aef2386ef9ed","Type":"ContainerStarted","Data":"ccfd67dbd462039cc8b3d1136ab8c277bcd8f42fc7db35d57267c478e921b2a1"}
Jul 06 21:17:06.892112 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:06.892075    2581 generic.go:356] "Generic (PLEG): container finished" podID="7b673fdb-9f97-4c99-ac99-aef2386ef9ed" containerID="b985b23cbfc61bff5af3bb25fe2a860583fb5046546aa7173c655e423f83b1c1" exitCode=0
Jul 06 21:17:06.892577 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:06.892161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpqt9" event={"ID":"7b673fdb-9f97-4c99-ac99-aef2386ef9ed","Type":"ContainerDied","Data":"b985b23cbfc61bff5af3bb25fe2a860583fb5046546aa7173c655e423f83b1c1"}
Jul 06 21:17:07.896733 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:07.896692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpqt9" event={"ID":"7b673fdb-9f97-4c99-ac99-aef2386ef9ed","Type":"ContainerStarted","Data":"021c518be8612b4a35fdf08f4deacea6e1a3caf5531d6cefead7dd49702f3d04"}
Jul 06 21:17:07.897108 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:07.896753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpqt9" event={"ID":"7b673fdb-9f97-4c99-ac99-aef2386ef9ed","Type":"ContainerStarted","Data":"06b67c364a7936bb8e3490ed663ddf298b32286df7927a0aa1be3aee5fce180e"}
Jul 06 21:17:07.918269 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:07.918201    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xpqt9" podStartSLOduration=2.752065823 podStartE2EDuration="3.918184504s" totalImagesPullingTime="1.102586219s" totalInitContainerRuntime="63.532462ms" isStatefulPod=true podCreationTimestamp="2026-07-06 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:17:07.917653526 +0000 UTC m=+264.911752689" watchObservedRunningTime="2026-07-06 21:17:07.918184504 +0000 UTC m=+264.912283663"
Jul 06 21:17:19.111187 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:19.111147    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd" podUID="b18744f6-a13d-4da8-addf-9e85256d122b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 06 21:17:25.569416 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:17:25.569363    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9gs9l" podUID="8a932825-f754-47ac-80b8-9d1e9a0b7d47"
Jul 06 21:17:25.569416 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:17:25.569363    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct" podUID="eda8e131-c168-47dc-9b65-591afe7cedc9"
Jul 06 21:17:25.949100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:25.949070    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:17:25.949100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:25.949098    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:17:28.750292 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:28.750262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:17:28.750292 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:28.750304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:17:28.752614 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:28.752581    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a932825-f754-47ac-80b8-9d1e9a0b7d47-metrics-tls\") pod \"dns-default-9gs9l\" (UID: \"8a932825-f754-47ac-80b8-9d1e9a0b7d47\") " pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:17:28.752730 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:28.752636    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"image-registry-84f76685fb-ht8ct\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") " pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:17:28.953110 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:28.953081    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-45jgn\""
Jul 06 21:17:28.953292 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:28.953082    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7xhcl\""
Jul 06 21:17:28.959846 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:28.959830    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:17:28.959979 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:28.959931    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:17:29.110804 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.110774    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9gs9l"]
Jul 06 21:17:29.111647 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.111621    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd" podUID="b18744f6-a13d-4da8-addf-9e85256d122b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 06 21:17:29.113107 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:17:29.113080    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a932825_f754_47ac_80b8_9d1e9a0b7d47.slice/crio-f9be43852745cd30467d33bc41be94c255f62b35067745181eb29cdbd5ce4d2d WatchSource:0}: Error finding container f9be43852745cd30467d33bc41be94c255f62b35067745181eb29cdbd5ce4d2d: Status 404 returned error can't find the container with id f9be43852745cd30467d33bc41be94c255f62b35067745181eb29cdbd5ce4d2d
Jul 06 21:17:29.130647 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.130620    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f76685fb-ht8ct"]
Jul 06 21:17:29.133710 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:17:29.133690    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda8e131_c168_47dc_9b65_591afe7cedc9.slice/crio-bcd7691794ff6dd5b84c2a1c3cbc7e7bfe76e81fb59e986e0df777ec8090f645 WatchSource:0}: Error finding container bcd7691794ff6dd5b84c2a1c3cbc7e7bfe76e81fb59e986e0df777ec8090f645: Status 404 returned error can't find the container with id bcd7691794ff6dd5b84c2a1c3cbc7e7bfe76e81fb59e986e0df777ec8090f645
Jul 06 21:17:29.965572 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.965432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct" event={"ID":"eda8e131-c168-47dc-9b65-591afe7cedc9","Type":"ContainerStarted","Data":"b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db"}
Jul 06 21:17:29.965572 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.965494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct" event={"ID":"eda8e131-c168-47dc-9b65-591afe7cedc9","Type":"ContainerStarted","Data":"bcd7691794ff6dd5b84c2a1c3cbc7e7bfe76e81fb59e986e0df777ec8090f645"}
Jul 06 21:17:29.966103 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.965590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:17:29.967097 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.967070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9gs9l" event={"ID":"8a932825-f754-47ac-80b8-9d1e9a0b7d47","Type":"ContainerStarted","Data":"f9be43852745cd30467d33bc41be94c255f62b35067745181eb29cdbd5ce4d2d"}
Jul 06 21:17:29.968878 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.968850    2581 generic.go:356] "Generic (PLEG): container finished" podID="20a3bd69-8cff-441d-a149-5d80353eade0" containerID="4ef922f5938dd437d451df00de785b54315d0f7922516e7fbc7369c1e79e813f" exitCode=1
Jul 06 21:17:29.968996 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.968917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv" event={"ID":"20a3bd69-8cff-441d-a149-5d80353eade0","Type":"ContainerDied","Data":"4ef922f5938dd437d451df00de785b54315d0f7922516e7fbc7369c1e79e813f"}
Jul 06 21:17:29.968996 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.968974    2581 scope.go:117] "RemoveContainer" containerID="5be53ae8e39d334afdcfa92eb550590eb302db86b817c0e10e9a9eb5be7179fe"
Jul 06 21:17:29.969296 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.969280    2581 scope.go:117] "RemoveContainer" containerID="4ef922f5938dd437d451df00de785b54315d0f7922516e7fbc7369c1e79e813f"
Jul 06 21:17:29.969485 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:17:29.969462    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-5f6b9f8f8d-24lkv_open-cluster-management-agent-addon(20a3bd69-8cff-441d-a149-5d80353eade0)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv" podUID="20a3bd69-8cff-441d-a149-5d80353eade0"
Jul 06 21:17:29.971025 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.971006    2581 generic.go:356] "Generic (PLEG): container finished" podID="df78cde0-2290-4097-9ebb-54fd8f3fbca0" containerID="2f620de1b3210035661d0562134a7afe3dc9239ca9500d7cd77954b734141012" exitCode=255
Jul 06 21:17:29.971109 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.971077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg" event={"ID":"df78cde0-2290-4097-9ebb-54fd8f3fbca0","Type":"ContainerDied","Data":"2f620de1b3210035661d0562134a7afe3dc9239ca9500d7cd77954b734141012"}
Jul 06 21:17:29.971368 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.971355    2581 scope.go:117] "RemoveContainer" containerID="2f620de1b3210035661d0562134a7afe3dc9239ca9500d7cd77954b734141012"
Jul 06 21:17:29.971502 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:17:29.971479    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-56657dc854-wg7cg_open-cluster-management-agent-addon(df78cde0-2290-4097-9ebb-54fd8f3fbca0)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg" podUID="df78cde0-2290-4097-9ebb-54fd8f3fbca0"
Jul 06 21:17:29.979506 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.979398    2581 scope.go:117] "RemoveContainer" containerID="46958524c267d38a7505bc6dd2fd44f13ef27b4d86ef697976eaee7bba5fd065"
Jul 06 21:17:29.985569 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:29.985524    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct" podStartSLOduration=284.98550662 podStartE2EDuration="4m44.98550662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:17:29.985423382 +0000 UTC m=+286.979522554" watchObservedRunningTime="2026-07-06 21:17:29.98550662 +0000 UTC m=+286.979605781"
Jul 06 21:17:30.597156 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:30.597102    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:17:30.976717 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:30.976684    2581 scope.go:117] "RemoveContainer" containerID="4ef922f5938dd437d451df00de785b54315d0f7922516e7fbc7369c1e79e813f"
Jul 06 21:17:30.977171 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:17:30.976898    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-5f6b9f8f8d-24lkv_open-cluster-management-agent-addon(20a3bd69-8cff-441d-a149-5d80353eade0)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv" podUID="20a3bd69-8cff-441d-a149-5d80353eade0"
Jul 06 21:17:31.985791 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:31.985761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9gs9l" event={"ID":"8a932825-f754-47ac-80b8-9d1e9a0b7d47","Type":"ContainerStarted","Data":"a5e7de5c3401b0e0327b3119df0e25015ccf6d9771a93546781e73ea73aa673e"}
Jul 06 21:17:31.986170 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:31.985808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9gs9l" event={"ID":"8a932825-f754-47ac-80b8-9d1e9a0b7d47","Type":"ContainerStarted","Data":"9a283215da80ffd4876dd5aac2aee36a1f3cff35057b016e4debbcb7eb01bd84"}
Jul 06 21:17:31.986170 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:31.985883    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:17:32.006190 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:32.006143    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9gs9l" podStartSLOduration=251.896851655 podStartE2EDuration="4m14.006125588s" totalImagesPullingTime="2.109273933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:17:32.005388156 +0000 UTC m=+288.999487326" watchObservedRunningTime="2026-07-06 21:17:32.006125588 +0000 UTC m=+289.000224747"
Jul 06 21:17:39.084925 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:39.084871    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg"
Jul 06 21:17:39.085369 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:39.085208    2581 scope.go:117] "RemoveContainer" containerID="2f620de1b3210035661d0562134a7afe3dc9239ca9500d7cd77954b734141012"
Jul 06 21:17:39.085411 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:17:39.085385    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-56657dc854-wg7cg_open-cluster-management-agent-addon(df78cde0-2290-4097-9ebb-54fd8f3fbca0)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg" podUID="df78cde0-2290-4097-9ebb-54fd8f3fbca0"
Jul 06 21:17:39.094972 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:39.094950    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:17:39.095280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:39.095268    2581 scope.go:117] "RemoveContainer" containerID="4ef922f5938dd437d451df00de785b54315d0f7922516e7fbc7369c1e79e813f"
Jul 06 21:17:39.095410 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:17:39.095394    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-5f6b9f8f8d-24lkv_open-cluster-management-agent-addon(20a3bd69-8cff-441d-a149-5d80353eade0)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv" podUID="20a3bd69-8cff-441d-a149-5d80353eade0"
Jul 06 21:17:39.111371 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:39.111339    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd" podUID="b18744f6-a13d-4da8-addf-9e85256d122b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 06 21:17:39.111515 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:39.111404    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd"
Jul 06 21:17:39.111896 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:39.111862    2581 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"600213b6dc88eedb28b52f59c441a741001d1eff482079b128f6682f11086c2f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 06 21:17:39.111988 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:39.111918    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd" podUID="b18744f6-a13d-4da8-addf-9e85256d122b" containerName="service-proxy" containerID="cri-o://600213b6dc88eedb28b52f59c441a741001d1eff482079b128f6682f11086c2f" gracePeriod=30
Jul 06 21:17:40.013091 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:40.013055    2581 generic.go:356] "Generic (PLEG): container finished" podID="b18744f6-a13d-4da8-addf-9e85256d122b" containerID="600213b6dc88eedb28b52f59c441a741001d1eff482079b128f6682f11086c2f" exitCode=2
Jul 06 21:17:40.013300 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:40.013112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd" event={"ID":"b18744f6-a13d-4da8-addf-9e85256d122b","Type":"ContainerDied","Data":"600213b6dc88eedb28b52f59c441a741001d1eff482079b128f6682f11086c2f"}
Jul 06 21:17:40.013300 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:40.013156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-659f984cd4-kxlrd" event={"ID":"b18744f6-a13d-4da8-addf-9e85256d122b","Type":"ContainerStarted","Data":"5e34652abd01dff00024e8d9ac2622db2b12199d2cf669fd819fff4422b12de7"}
Jul 06 21:17:41.991951 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:41.991920    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9gs9l"
Jul 06 21:17:43.747456 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:43.747431    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 21:17:48.965701 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:48.965658    2581 patch_prober.go:28] interesting pod/image-registry-84f76685fb-ht8ct container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 06 21:17:48.966111 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:48.965752    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct" podUID="eda8e131-c168-47dc-9b65-591afe7cedc9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:17:50.872754 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:50.872720    2581 scope.go:117] "RemoveContainer" containerID="2f620de1b3210035661d0562134a7afe3dc9239ca9500d7cd77954b734141012"
Jul 06 21:17:50.983124 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:50.983098    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:17:51.052553 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:51.052520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56657dc854-wg7cg" event={"ID":"df78cde0-2290-4097-9ebb-54fd8f3fbca0","Type":"ContainerStarted","Data":"a57833af2bbff4ef41461c22ba0eb802cd389c15ed12affe8427b3f75d372d6b"}
Jul 06 21:17:51.873360 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:51.873332    2581 scope.go:117] "RemoveContainer" containerID="4ef922f5938dd437d451df00de785b54315d0f7922516e7fbc7369c1e79e813f"
Jul 06 21:17:52.058100 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:52.058066    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv" event={"ID":"20a3bd69-8cff-441d-a149-5d80353eade0","Type":"ContainerStarted","Data":"bc73dbc78b9278319a9b5b6853167b22e028733c2a12bf3d7d06e9a8efba97b1"}
Jul 06 21:17:52.058572 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:52.058515    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:17:52.060011 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:52.059992    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f6b9f8f8d-24lkv"
Jul 06 21:17:57.337841 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:17:57.337809    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f76685fb-ht8ct"]
Jul 06 21:18:22.357162 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:22.357122    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct" podUID="eda8e131-c168-47dc-9b65-591afe7cedc9" containerName="registry" containerID="cri-o://b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db" gracePeriod=30
Jul 06 21:18:23.594148 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.594123    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:18:23.633930 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.633901    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eda8e131-c168-47dc-9b65-591afe7cedc9-trusted-ca\") pod \"eda8e131-c168-47dc-9b65-591afe7cedc9\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") "
Jul 06 21:18:23.634108 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.633946    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eda8e131-c168-47dc-9b65-591afe7cedc9-ca-trust-extracted\") pod \"eda8e131-c168-47dc-9b65-591afe7cedc9\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") "
Jul 06 21:18:23.634108 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.633989    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwt7h\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-kube-api-access-mwt7h\") pod \"eda8e131-c168-47dc-9b65-591afe7cedc9\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") "
Jul 06 21:18:23.634108 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.634028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-bound-sa-token\") pod \"eda8e131-c168-47dc-9b65-591afe7cedc9\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") "
Jul 06 21:18:23.634108 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.634053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eda8e131-c168-47dc-9b65-591afe7cedc9-installation-pull-secrets\") pod \"eda8e131-c168-47dc-9b65-591afe7cedc9\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") "
Jul 06 21:18:23.634108 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.634081    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-certificates\") pod \"eda8e131-c168-47dc-9b65-591afe7cedc9\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") "
Jul 06 21:18:23.634108 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.634105    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eda8e131-c168-47dc-9b65-591afe7cedc9-image-registry-private-configuration\") pod \"eda8e131-c168-47dc-9b65-591afe7cedc9\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") "
Jul 06 21:18:23.634415 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.634140    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") pod \"eda8e131-c168-47dc-9b65-591afe7cedc9\" (UID: \"eda8e131-c168-47dc-9b65-591afe7cedc9\") "
Jul 06 21:18:23.634472 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.634448    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda8e131-c168-47dc-9b65-591afe7cedc9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "eda8e131-c168-47dc-9b65-591afe7cedc9" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:18:23.634687 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.634642    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "eda8e131-c168-47dc-9b65-591afe7cedc9" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:18:23.636953 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.636921    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-kube-api-access-mwt7h" (OuterVolumeSpecName: "kube-api-access-mwt7h") pod "eda8e131-c168-47dc-9b65-591afe7cedc9" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9"). InnerVolumeSpecName "kube-api-access-mwt7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:18:23.636953 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.636931    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda8e131-c168-47dc-9b65-591afe7cedc9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "eda8e131-c168-47dc-9b65-591afe7cedc9" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:18:23.637146 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.637048    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "eda8e131-c168-47dc-9b65-591afe7cedc9" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:18:23.637146 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.637073    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda8e131-c168-47dc-9b65-591afe7cedc9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "eda8e131-c168-47dc-9b65-591afe7cedc9" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:18:23.637276 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.637155    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "eda8e131-c168-47dc-9b65-591afe7cedc9" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:18:23.643582 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.643550    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda8e131-c168-47dc-9b65-591afe7cedc9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "eda8e131-c168-47dc-9b65-591afe7cedc9" (UID: "eda8e131-c168-47dc-9b65-591afe7cedc9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:18:23.734875 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.734793    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eda8e131-c168-47dc-9b65-591afe7cedc9-trusted-ca\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:18:23.734875 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.734820    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eda8e131-c168-47dc-9b65-591afe7cedc9-ca-trust-extracted\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:18:23.734875 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.734829    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwt7h\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-kube-api-access-mwt7h\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:18:23.734875 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.734841    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-bound-sa-token\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:18:23.734875 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.734850    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eda8e131-c168-47dc-9b65-591afe7cedc9-installation-pull-secrets\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:18:23.734875 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.734859    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-certificates\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:18:23.734875 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.734868    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eda8e131-c168-47dc-9b65-591afe7cedc9-image-registry-private-configuration\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:18:23.734875 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:23.734877    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda8e131-c168-47dc-9b65-591afe7cedc9-registry-tls\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:18:24.164553 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:24.164514    2581 generic.go:356] "Generic (PLEG): container finished" podID="eda8e131-c168-47dc-9b65-591afe7cedc9" containerID="b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db" exitCode=0
Jul 06 21:18:24.164727 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:24.164578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct" event={"ID":"eda8e131-c168-47dc-9b65-591afe7cedc9","Type":"ContainerDied","Data":"b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db"}
Jul 06 21:18:24.164727 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:24.164584    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct"
Jul 06 21:18:24.164727 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:24.164625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f76685fb-ht8ct" event={"ID":"eda8e131-c168-47dc-9b65-591afe7cedc9","Type":"ContainerDied","Data":"bcd7691794ff6dd5b84c2a1c3cbc7e7bfe76e81fb59e986e0df777ec8090f645"}
Jul 06 21:18:24.164727 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:24.164652    2581 scope.go:117] "RemoveContainer" containerID="b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db"
Jul 06 21:18:24.172269 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:24.172251    2581 scope.go:117] "RemoveContainer" containerID="b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db"
Jul 06 21:18:24.172528 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:18:24.172503    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db\": container with ID starting with b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db not found: ID does not exist" containerID="b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db"
Jul 06 21:18:24.172608 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:24.172538    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db"} err="failed to get container status \"b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db\": rpc error: code = NotFound desc = could not find container \"b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db\": container with ID starting with b92d5d524d2b27aa51db06a31fe1138d1fae017b90a7f893e6783196b9f719db not found: ID does not exist"
Jul 06 21:18:24.181734 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:24.181713    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f76685fb-ht8ct"]
Jul 06 21:18:24.185618 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:24.185599    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84f76685fb-ht8ct"]
Jul 06 21:18:25.877981 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:18:25.877949    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eda8e131-c168-47dc-9b65-591afe7cedc9" path="/var/lib/kubelet/pods/eda8e131-c168-47dc-9b65-591afe7cedc9/volumes"
Jul 06 21:22:33.914839 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.914806    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7ccfc88547-mpwzg"]
Jul 06 21:22:33.915370 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.915052    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eda8e131-c168-47dc-9b65-591afe7cedc9" containerName="registry"
Jul 06 21:22:33.915370 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.915067    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eda8e131-c168-47dc-9b65-591afe7cedc9" containerName="registry"
Jul 06 21:22:33.915370 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.915126    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eda8e131-c168-47dc-9b65-591afe7cedc9" containerName="registry"
Jul 06 21:22:33.916961 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.916942    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg"
Jul 06 21:22:33.919633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.919612    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 06 21:22:33.919853 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.919823    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 06 21:22:33.920515 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.920496    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 06 21:22:33.920599 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.920550    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-7sd78\""
Jul 06 21:22:33.930873 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.930854    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7ccfc88547-mpwzg"]
Jul 06 21:22:33.963614 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.963584    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-qx9x4"]
Jul 06 21:22:33.965929 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.965908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qx9x4"
Jul 06 21:22:33.968710 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.968688    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 06 21:22:33.968808 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.968688    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6g67l\""
Jul 06 21:22:33.979096 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:33.979076    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qx9x4"]
Jul 06 21:22:34.027140 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.027110    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbm75\" (UniqueName: \"kubernetes.io/projected/6772628d-9c5a-42af-a34c-088a2a3503fb-kube-api-access-zbm75\") pod \"seaweedfs-64568bcd49-qx9x4\" (UID: \"6772628d-9c5a-42af-a34c-088a2a3503fb\") " pod="kserve/seaweedfs-64568bcd49-qx9x4"
Jul 06 21:22:34.027140 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.027149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rf8v\" (UniqueName: \"kubernetes.io/projected/0b5de1d8-1b14-4559-8bb0-fa2325469aad-kube-api-access-8rf8v\") pod \"kserve-controller-manager-7ccfc88547-mpwzg\" (UID: \"0b5de1d8-1b14-4559-8bb0-fa2325469aad\") " pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg"
Jul 06 21:22:34.027412 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.027174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b5de1d8-1b14-4559-8bb0-fa2325469aad-cert\") pod \"kserve-controller-manager-7ccfc88547-mpwzg\" (UID: \"0b5de1d8-1b14-4559-8bb0-fa2325469aad\") " pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg"
Jul 06 21:22:34.027412 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.027297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6772628d-9c5a-42af-a34c-088a2a3503fb-data\") pod \"seaweedfs-64568bcd49-qx9x4\" (UID: \"6772628d-9c5a-42af-a34c-088a2a3503fb\") " pod="kserve/seaweedfs-64568bcd49-qx9x4"
Jul 06 21:22:34.128298 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.128265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b5de1d8-1b14-4559-8bb0-fa2325469aad-cert\") pod \"kserve-controller-manager-7ccfc88547-mpwzg\" (UID: \"0b5de1d8-1b14-4559-8bb0-fa2325469aad\") " pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg"
Jul 06 21:22:34.128497 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.128311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6772628d-9c5a-42af-a34c-088a2a3503fb-data\") pod \"seaweedfs-64568bcd49-qx9x4\" (UID: \"6772628d-9c5a-42af-a34c-088a2a3503fb\") " pod="kserve/seaweedfs-64568bcd49-qx9x4"
Jul 06 21:22:34.128497 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.128351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbm75\" (UniqueName: \"kubernetes.io/projected/6772628d-9c5a-42af-a34c-088a2a3503fb-kube-api-access-zbm75\") pod \"seaweedfs-64568bcd49-qx9x4\" (UID: \"6772628d-9c5a-42af-a34c-088a2a3503fb\") " pod="kserve/seaweedfs-64568bcd49-qx9x4"
Jul 06 21:22:34.128497 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.128372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rf8v\" (UniqueName: \"kubernetes.io/projected/0b5de1d8-1b14-4559-8bb0-fa2325469aad-kube-api-access-8rf8v\") pod \"kserve-controller-manager-7ccfc88547-mpwzg\" (UID: \"0b5de1d8-1b14-4559-8bb0-fa2325469aad\") " pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg"
Jul 06 21:22:34.128740 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.128717    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6772628d-9c5a-42af-a34c-088a2a3503fb-data\") pod \"seaweedfs-64568bcd49-qx9x4\" (UID: \"6772628d-9c5a-42af-a34c-088a2a3503fb\") " pod="kserve/seaweedfs-64568bcd49-qx9x4"
Jul 06 21:22:34.130652 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.130634    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b5de1d8-1b14-4559-8bb0-fa2325469aad-cert\") pod \"kserve-controller-manager-7ccfc88547-mpwzg\" (UID: \"0b5de1d8-1b14-4559-8bb0-fa2325469aad\") " pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg"
Jul 06 21:22:34.138072 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.138044    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbm75\" (UniqueName: \"kubernetes.io/projected/6772628d-9c5a-42af-a34c-088a2a3503fb-kube-api-access-zbm75\") pod \"seaweedfs-64568bcd49-qx9x4\" (UID: \"6772628d-9c5a-42af-a34c-088a2a3503fb\") " pod="kserve/seaweedfs-64568bcd49-qx9x4"
Jul 06 21:22:34.138169 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.138109    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rf8v\" (UniqueName: \"kubernetes.io/projected/0b5de1d8-1b14-4559-8bb0-fa2325469aad-kube-api-access-8rf8v\") pod \"kserve-controller-manager-7ccfc88547-mpwzg\" (UID: \"0b5de1d8-1b14-4559-8bb0-fa2325469aad\") " pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg"
Jul 06 21:22:34.227579 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.227493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg"
Jul 06 21:22:34.274797 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.274221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qx9x4"
Jul 06 21:22:34.365504 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.365447    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7ccfc88547-mpwzg"]
Jul 06 21:22:34.368609 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:22:34.368577    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5de1d8_1b14_4559_8bb0_fa2325469aad.slice/crio-9412240c841e82f44b0477f90fd9b609b774e2138704b1d7bd741b8bb78800dd WatchSource:0}: Error finding container 9412240c841e82f44b0477f90fd9b609b774e2138704b1d7bd741b8bb78800dd: Status 404 returned error can't find the container with id 9412240c841e82f44b0477f90fd9b609b774e2138704b1d7bd741b8bb78800dd
Jul 06 21:22:34.369774 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.369757    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:22:34.405698 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.405667    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qx9x4"]
Jul 06 21:22:34.408957 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:22:34.408935    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6772628d_9c5a_42af_a34c_088a2a3503fb.slice/crio-1ff25a19b208c3258609c8df184de7f46ddf319ca6229d7e0e38e8719fca8f09 WatchSource:0}: Error finding container 1ff25a19b208c3258609c8df184de7f46ddf319ca6229d7e0e38e8719fca8f09: Status 404 returned error can't find the container with id 1ff25a19b208c3258609c8df184de7f46ddf319ca6229d7e0e38e8719fca8f09
Jul 06 21:22:34.950768 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.950735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qx9x4" event={"ID":"6772628d-9c5a-42af-a34c-088a2a3503fb","Type":"ContainerStarted","Data":"1ff25a19b208c3258609c8df184de7f46ddf319ca6229d7e0e38e8719fca8f09"}
Jul 06 21:22:34.951800 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:34.951776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg" event={"ID":"0b5de1d8-1b14-4559-8bb0-fa2325469aad","Type":"ContainerStarted","Data":"9412240c841e82f44b0477f90fd9b609b774e2138704b1d7bd741b8bb78800dd"}
Jul 06 21:22:38.966449 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:38.966415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qx9x4" event={"ID":"6772628d-9c5a-42af-a34c-088a2a3503fb","Type":"ContainerStarted","Data":"c58b0550b370ce25a76b784848ac6ff8561cc5f01781e73eb3392d4458f31a43"}
Jul 06 21:22:38.966883 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:38.966516    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-qx9x4"
Jul 06 21:22:38.967902 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:38.967880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg" event={"ID":"0b5de1d8-1b14-4559-8bb0-fa2325469aad","Type":"ContainerStarted","Data":"d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791"}
Jul 06 21:22:38.968003 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:38.967975    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg"
Jul 06 21:22:38.988356 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:38.988312    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-qx9x4" podStartSLOduration=2.203558609 podStartE2EDuration="5.988300177s" totalImagesPullingTime="3.784741568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:22:38.98636594 +0000 UTC m=+595.980465101" watchObservedRunningTime="2026-07-06 21:22:38.988300177 +0000 UTC m=+595.982399338"
Jul 06 21:22:39.014127 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:39.014082    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg" podStartSLOduration=2.274487894 podStartE2EDuration="6.014067546s" totalImagesPullingTime="3.739579652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:22:39.013443008 +0000 UTC m=+596.007542180" watchObservedRunningTime="2026-07-06 21:22:39.014067546 +0000 UTC m=+596.008166706"
Jul 06 21:22:44.973613 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:22:44.973581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-qx9x4"
Jul 06 21:23:09.561642 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.561561    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7ccfc88547-mpwzg"]
Jul 06 21:23:09.562129 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.561779    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg" podUID="0b5de1d8-1b14-4559-8bb0-fa2325469aad" containerName="manager" containerID="cri-o://d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791" gracePeriod=10
Jul 06 21:23:09.566805 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.566779    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg"
Jul 06 21:23:09.580410 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.580387    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7ccfc88547-c7vz8"]
Jul 06 21:23:09.583201 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.583183    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8"
Jul 06 21:23:09.593571 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.593549    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7ccfc88547-c7vz8"]
Jul 06 21:23:09.677084 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.677050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsp4k\" (UniqueName: \"kubernetes.io/projected/d315926c-34e4-4994-84bc-d80cf47ebcf9-kube-api-access-gsp4k\") pod \"kserve-controller-manager-7ccfc88547-c7vz8\" (UID: \"d315926c-34e4-4994-84bc-d80cf47ebcf9\") " pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8"
Jul 06 21:23:09.677256 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.677162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d315926c-34e4-4994-84bc-d80cf47ebcf9-cert\") pod \"kserve-controller-manager-7ccfc88547-c7vz8\" (UID: \"d315926c-34e4-4994-84bc-d80cf47ebcf9\") " pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8"
Jul 06 21:23:09.778282 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.778221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsp4k\" (UniqueName: \"kubernetes.io/projected/d315926c-34e4-4994-84bc-d80cf47ebcf9-kube-api-access-gsp4k\") pod \"kserve-controller-manager-7ccfc88547-c7vz8\" (UID: \"d315926c-34e4-4994-84bc-d80cf47ebcf9\") " pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8"
Jul 06 21:23:09.778457 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.778319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d315926c-34e4-4994-84bc-d80cf47ebcf9-cert\") pod \"kserve-controller-manager-7ccfc88547-c7vz8\" (UID: \"d315926c-34e4-4994-84bc-d80cf47ebcf9\") " pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8"
Jul 06 21:23:09.780489 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.780466    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d315926c-34e4-4994-84bc-d80cf47ebcf9-cert\") pod \"kserve-controller-manager-7ccfc88547-c7vz8\" (UID: \"d315926c-34e4-4994-84bc-d80cf47ebcf9\") " pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8"
Jul 06 21:23:09.786252 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.786204    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsp4k\" (UniqueName: \"kubernetes.io/projected/d315926c-34e4-4994-84bc-d80cf47ebcf9-kube-api-access-gsp4k\") pod \"kserve-controller-manager-7ccfc88547-c7vz8\" (UID: \"d315926c-34e4-4994-84bc-d80cf47ebcf9\") " pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8"
Jul 06 21:23:09.800384 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.800363    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg"
Jul 06 21:23:09.878681 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.878606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b5de1d8-1b14-4559-8bb0-fa2325469aad-cert\") pod \"0b5de1d8-1b14-4559-8bb0-fa2325469aad\" (UID: \"0b5de1d8-1b14-4559-8bb0-fa2325469aad\") "
Jul 06 21:23:09.878681 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.878635    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rf8v\" (UniqueName: \"kubernetes.io/projected/0b5de1d8-1b14-4559-8bb0-fa2325469aad-kube-api-access-8rf8v\") pod \"0b5de1d8-1b14-4559-8bb0-fa2325469aad\" (UID: \"0b5de1d8-1b14-4559-8bb0-fa2325469aad\") "
Jul 06 21:23:09.880792 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.880756    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5de1d8-1b14-4559-8bb0-fa2325469aad-cert" (OuterVolumeSpecName: "cert") pod "0b5de1d8-1b14-4559-8bb0-fa2325469aad" (UID: "0b5de1d8-1b14-4559-8bb0-fa2325469aad"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:23:09.880904 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.880837    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5de1d8-1b14-4559-8bb0-fa2325469aad-kube-api-access-8rf8v" (OuterVolumeSpecName: "kube-api-access-8rf8v") pod "0b5de1d8-1b14-4559-8bb0-fa2325469aad" (UID: "0b5de1d8-1b14-4559-8bb0-fa2325469aad"). InnerVolumeSpecName "kube-api-access-8rf8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:23:09.958184 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.958136    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8"
Jul 06 21:23:09.979642 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.979613    2581 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b5de1d8-1b14-4559-8bb0-fa2325469aad-cert\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:23:09.979642 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:09.979644    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rf8v\" (UniqueName: \"kubernetes.io/projected/0b5de1d8-1b14-4559-8bb0-fa2325469aad-kube-api-access-8rf8v\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:23:10.069855 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:10.069825    2581 generic.go:356] "Generic (PLEG): container finished" podID="0b5de1d8-1b14-4559-8bb0-fa2325469aad" containerID="d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791" exitCode=0
Jul 06 21:23:10.070017 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:10.069870    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg"
Jul 06 21:23:10.070017 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:10.069904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg" event={"ID":"0b5de1d8-1b14-4559-8bb0-fa2325469aad","Type":"ContainerDied","Data":"d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791"}
Jul 06 21:23:10.070017 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:10.069959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7ccfc88547-mpwzg" event={"ID":"0b5de1d8-1b14-4559-8bb0-fa2325469aad","Type":"ContainerDied","Data":"9412240c841e82f44b0477f90fd9b609b774e2138704b1d7bd741b8bb78800dd"}
Jul 06 21:23:10.070017 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:10.069988    2581 scope.go:117] "RemoveContainer" containerID="d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791"
Jul 06 21:23:10.078467 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:10.078447    2581 scope.go:117] "RemoveContainer" containerID="d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791"
Jul 06 21:23:10.078747 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:23:10.078722    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791\": container with ID starting with d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791 not found: ID does not exist" containerID="d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791"
Jul 06 21:23:10.078828 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:10.078754    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791"} err="failed to get container status \"d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791\": rpc error: code = NotFound desc = could not find container \"d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791\": container with ID starting with d630fd95ab6dab899652dc465d077dd0bbf1a44913c4fc9d98d09af49d7ad791 not found: ID does not exist"
Jul 06 21:23:10.083407 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:10.083372    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7ccfc88547-c7vz8"]
Jul 06 21:23:10.086411 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:23:10.086382    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd315926c_34e4_4994_84bc_d80cf47ebcf9.slice/crio-dde5a2b033e8875519216ba0a48c5b3933512aee9d07b3936e6b8306b4467c4d WatchSource:0}: Error finding container dde5a2b033e8875519216ba0a48c5b3933512aee9d07b3936e6b8306b4467c4d: Status 404 returned error can't find the container with id dde5a2b033e8875519216ba0a48c5b3933512aee9d07b3936e6b8306b4467c4d
Jul 06 21:23:10.091678 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:10.091659    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7ccfc88547-mpwzg"]
Jul 06 21:23:10.094440 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:10.094421    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7ccfc88547-mpwzg"]
Jul 06 21:23:11.075014 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:11.074928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8" event={"ID":"d315926c-34e4-4994-84bc-d80cf47ebcf9","Type":"ContainerStarted","Data":"03e3082839e871c178355a1224bbd6b2fc1e8a587ba751b3f2ab54fccdccfd79"}
Jul 06 21:23:11.075014 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:11.074993    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8"
Jul 06 21:23:11.075014 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:11.075015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8" event={"ID":"d315926c-34e4-4994-84bc-d80cf47ebcf9","Type":"ContainerStarted","Data":"dde5a2b033e8875519216ba0a48c5b3933512aee9d07b3936e6b8306b4467c4d"}
Jul 06 21:23:11.091330 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:11.091287    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8" podStartSLOduration=1.462168933 podStartE2EDuration="2.091272281s" totalImagesPullingTime="629.103348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:23:11.090718082 +0000 UTC m=+628.084817257" watchObservedRunningTime="2026-07-06 21:23:11.091272281 +0000 UTC m=+628.085371458"
Jul 06 21:23:11.877576 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:11.877541    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5de1d8-1b14-4559-8bb0-fa2325469aad" path="/var/lib/kubelet/pods/0b5de1d8-1b14-4559-8bb0-fa2325469aad/volumes"
Jul 06 21:23:42.084796 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:42.084766    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7ccfc88547-c7vz8"
Jul 06 21:23:43.248031 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.247995    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-67p6q"]
Jul 06 21:23:43.248420 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.248275    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5de1d8-1b14-4559-8bb0-fa2325469aad" containerName="manager"
Jul 06 21:23:43.248420 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.248293    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5de1d8-1b14-4559-8bb0-fa2325469aad" containerName="manager"
Jul 06 21:23:43.248420 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.248373    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5de1d8-1b14-4559-8bb0-fa2325469aad" containerName="manager"
Jul 06 21:23:43.251254 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.251221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-67p6q"
Jul 06 21:23:43.258421 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.258397    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-pkz4z\""
Jul 06 21:23:43.258612 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.258453    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 06 21:23:43.261823 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.261799    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-67p6q"]
Jul 06 21:23:43.264947 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.264921    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-mxfcs"]
Jul 06 21:23:43.267887 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.267864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-mxfcs"
Jul 06 21:23:43.269969 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.269946    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 06 21:23:43.270114 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.270049    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-pchqn\""
Jul 06 21:23:43.279365 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.279340    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-mxfcs"]
Jul 06 21:23:43.299682 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.299657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftdxc\" (UniqueName: \"kubernetes.io/projected/75978d84-5ef5-49f6-b486-ce2bb5826b7c-kube-api-access-ftdxc\") pod \"odh-model-controller-84b54b7f97-mxfcs\" (UID: \"75978d84-5ef5-49f6-b486-ce2bb5826b7c\") " pod="kserve/odh-model-controller-84b54b7f97-mxfcs"
Jul 06 21:23:43.299804 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.299702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56eed0bb-9b6d-4fc7-990a-7111497515f3-tls-certs\") pod \"model-serving-api-fd65d7d6-67p6q\" (UID: \"56eed0bb-9b6d-4fc7-990a-7111497515f3\") " pod="kserve/model-serving-api-fd65d7d6-67p6q"
Jul 06 21:23:43.299804 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.299722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75978d84-5ef5-49f6-b486-ce2bb5826b7c-cert\") pod \"odh-model-controller-84b54b7f97-mxfcs\" (UID: \"75978d84-5ef5-49f6-b486-ce2bb5826b7c\") " pod="kserve/odh-model-controller-84b54b7f97-mxfcs"
Jul 06 21:23:43.299804 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.299737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbtz\" (UniqueName: \"kubernetes.io/projected/56eed0bb-9b6d-4fc7-990a-7111497515f3-kube-api-access-jhbtz\") pod \"model-serving-api-fd65d7d6-67p6q\" (UID: \"56eed0bb-9b6d-4fc7-990a-7111497515f3\") " pod="kserve/model-serving-api-fd65d7d6-67p6q"
Jul 06 21:23:43.400435 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.400394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56eed0bb-9b6d-4fc7-990a-7111497515f3-tls-certs\") pod \"model-serving-api-fd65d7d6-67p6q\" (UID: \"56eed0bb-9b6d-4fc7-990a-7111497515f3\") " pod="kserve/model-serving-api-fd65d7d6-67p6q"
Jul 06 21:23:43.400435 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.400441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75978d84-5ef5-49f6-b486-ce2bb5826b7c-cert\") pod \"odh-model-controller-84b54b7f97-mxfcs\" (UID: \"75978d84-5ef5-49f6-b486-ce2bb5826b7c\") " pod="kserve/odh-model-controller-84b54b7f97-mxfcs"
Jul 06 21:23:43.400689 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.400466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbtz\" (UniqueName: \"kubernetes.io/projected/56eed0bb-9b6d-4fc7-990a-7111497515f3-kube-api-access-jhbtz\") pod \"model-serving-api-fd65d7d6-67p6q\" (UID: \"56eed0bb-9b6d-4fc7-990a-7111497515f3\") " pod="kserve/model-serving-api-fd65d7d6-67p6q"
Jul 06 21:23:43.400689 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.400507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdxc\" (UniqueName: \"kubernetes.io/projected/75978d84-5ef5-49f6-b486-ce2bb5826b7c-kube-api-access-ftdxc\") pod \"odh-model-controller-84b54b7f97-mxfcs\" (UID: \"75978d84-5ef5-49f6-b486-ce2bb5826b7c\") " pod="kserve/odh-model-controller-84b54b7f97-mxfcs"
Jul 06 21:23:43.400689 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:23:43.400565    2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 06 21:23:43.400689 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:23:43.400645    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75978d84-5ef5-49f6-b486-ce2bb5826b7c-cert podName:75978d84-5ef5-49f6-b486-ce2bb5826b7c nodeName:}" failed. No retries permitted until 2026-07-06 21:23:43.900624622 +0000 UTC m=+660.894723767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/75978d84-5ef5-49f6-b486-ce2bb5826b7c-cert") pod "odh-model-controller-84b54b7f97-mxfcs" (UID: "75978d84-5ef5-49f6-b486-ce2bb5826b7c") : secret "odh-model-controller-webhook-cert" not found
Jul 06 21:23:43.402894 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.402866    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56eed0bb-9b6d-4fc7-990a-7111497515f3-tls-certs\") pod \"model-serving-api-fd65d7d6-67p6q\" (UID: \"56eed0bb-9b6d-4fc7-990a-7111497515f3\") " pod="kserve/model-serving-api-fd65d7d6-67p6q"
Jul 06 21:23:43.409122 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.409099    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdxc\" (UniqueName: \"kubernetes.io/projected/75978d84-5ef5-49f6-b486-ce2bb5826b7c-kube-api-access-ftdxc\") pod \"odh-model-controller-84b54b7f97-mxfcs\" (UID: \"75978d84-5ef5-49f6-b486-ce2bb5826b7c\") " pod="kserve/odh-model-controller-84b54b7f97-mxfcs"
Jul 06 21:23:43.409552 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.409535    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbtz\" (UniqueName: \"kubernetes.io/projected/56eed0bb-9b6d-4fc7-990a-7111497515f3-kube-api-access-jhbtz\") pod \"model-serving-api-fd65d7d6-67p6q\" (UID: \"56eed0bb-9b6d-4fc7-990a-7111497515f3\") " pod="kserve/model-serving-api-fd65d7d6-67p6q"
Jul 06 21:23:43.568996 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.568905    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-67p6q"
Jul 06 21:23:43.693191 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.693152    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-67p6q"]
Jul 06 21:23:43.696449 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:23:43.696423    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56eed0bb_9b6d_4fc7_990a_7111497515f3.slice/crio-88374cf71b3b116095223e1f2e92ff29b2f434f28e6a467e52f84a48fc05f903 WatchSource:0}: Error finding container 88374cf71b3b116095223e1f2e92ff29b2f434f28e6a467e52f84a48fc05f903: Status 404 returned error can't find the container with id 88374cf71b3b116095223e1f2e92ff29b2f434f28e6a467e52f84a48fc05f903
Jul 06 21:23:43.904355 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.904321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75978d84-5ef5-49f6-b486-ce2bb5826b7c-cert\") pod \"odh-model-controller-84b54b7f97-mxfcs\" (UID: \"75978d84-5ef5-49f6-b486-ce2bb5826b7c\") " pod="kserve/odh-model-controller-84b54b7f97-mxfcs"
Jul 06 21:23:43.906566 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:43.906547    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75978d84-5ef5-49f6-b486-ce2bb5826b7c-cert\") pod \"odh-model-controller-84b54b7f97-mxfcs\" (UID: \"75978d84-5ef5-49f6-b486-ce2bb5826b7c\") " pod="kserve/odh-model-controller-84b54b7f97-mxfcs"
Jul 06 21:23:44.180569 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:44.180478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-67p6q" event={"ID":"56eed0bb-9b6d-4fc7-990a-7111497515f3","Type":"ContainerStarted","Data":"88374cf71b3b116095223e1f2e92ff29b2f434f28e6a467e52f84a48fc05f903"}
Jul 06 21:23:44.180703 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:44.180608    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-pchqn\""
Jul 06 21:23:44.188518 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:44.188498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-mxfcs"
Jul 06 21:23:44.316104 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:44.316072    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-mxfcs"]
Jul 06 21:23:44.319513 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:23:44.319482    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75978d84_5ef5_49f6_b486_ce2bb5826b7c.slice/crio-f271744790a1d5bbfd9acd28a824e1a886974fdecf4e8ab50cd35acd87551321 WatchSource:0}: Error finding container f271744790a1d5bbfd9acd28a824e1a886974fdecf4e8ab50cd35acd87551321: Status 404 returned error can't find the container with id f271744790a1d5bbfd9acd28a824e1a886974fdecf4e8ab50cd35acd87551321
Jul 06 21:23:45.186420 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:45.186341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-mxfcs" event={"ID":"75978d84-5ef5-49f6-b486-ce2bb5826b7c","Type":"ContainerStarted","Data":"f271744790a1d5bbfd9acd28a824e1a886974fdecf4e8ab50cd35acd87551321"}
Jul 06 21:23:46.194879 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:46.194410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-67p6q" event={"ID":"56eed0bb-9b6d-4fc7-990a-7111497515f3","Type":"ContainerStarted","Data":"bb08682b95f02df1b12af94e9bdfca9e7c2e6ae6ca619604cb7a3ffc0b8969a3"}
Jul 06 21:23:46.194879 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:46.194538    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-67p6q"
Jul 06 21:23:46.213575 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:46.213522    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-67p6q" podStartSLOduration=1.226864363 podStartE2EDuration="3.213507056s" totalImagesPullingTime="1.986642693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:23:46.211939002 +0000 UTC m=+663.206038162" watchObservedRunningTime="2026-07-06 21:23:46.213507056 +0000 UTC m=+663.207606215"
Jul 06 21:23:48.204326 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:48.204285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-mxfcs" event={"ID":"75978d84-5ef5-49f6-b486-ce2bb5826b7c","Type":"ContainerStarted","Data":"45a14fa5da90cf01eae5b18fb409e5d065305e17b8d171cb88b4fd016075c377"}
Jul 06 21:23:48.204693 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:48.204437    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-mxfcs"
Jul 06 21:23:48.220005 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:48.219960    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-mxfcs" podStartSLOduration=2.259373879 podStartE2EDuration="5.21994758s" totalImagesPullingTime="2.960573701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:23:48.218563779 +0000 UTC m=+665.212662938" watchObservedRunningTime="2026-07-06 21:23:48.21994758 +0000 UTC m=+665.214046740"
Jul 06 21:23:57.204393 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:57.204363    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-67p6q"
Jul 06 21:23:59.210426 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:23:59.210396    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-mxfcs"
Jul 06 21:25:46.416542 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.416510    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"]
Jul 06 21:25:46.419592 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.419574    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"
Jul 06 21:25:46.421932 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.421912    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 06 21:25:46.422036 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.421911    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 06 21:25:46.429089 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.429060    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"]
Jul 06 21:25:46.477364 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.477326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prhdh\" (UniqueName: \"kubernetes.io/projected/7a8504e5-800b-429c-ba06-880a996e40ae-kube-api-access-prhdh\") pod \"seaweedfs-tls-serving-67b7477c6d-f8xfx\" (UID: \"7a8504e5-800b-429c-ba06-880a996e40ae\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"
Jul 06 21:25:46.477364 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.477370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/7a8504e5-800b-429c-ba06-880a996e40ae-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-f8xfx\" (UID: \"7a8504e5-800b-429c-ba06-880a996e40ae\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"
Jul 06 21:25:46.477582 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.477433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7a8504e5-800b-429c-ba06-880a996e40ae-data\") pod \"seaweedfs-tls-serving-67b7477c6d-f8xfx\" (UID: \"7a8504e5-800b-429c-ba06-880a996e40ae\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"
Jul 06 21:25:46.578612 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.578567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prhdh\" (UniqueName: \"kubernetes.io/projected/7a8504e5-800b-429c-ba06-880a996e40ae-kube-api-access-prhdh\") pod \"seaweedfs-tls-serving-67b7477c6d-f8xfx\" (UID: \"7a8504e5-800b-429c-ba06-880a996e40ae\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"
Jul 06 21:25:46.578785 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.578626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/7a8504e5-800b-429c-ba06-880a996e40ae-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-f8xfx\" (UID: \"7a8504e5-800b-429c-ba06-880a996e40ae\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"
Jul 06 21:25:46.578785 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.578665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7a8504e5-800b-429c-ba06-880a996e40ae-data\") pod \"seaweedfs-tls-serving-67b7477c6d-f8xfx\" (UID: \"7a8504e5-800b-429c-ba06-880a996e40ae\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"
Jul 06 21:25:46.578922 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:25:46.578802    2581 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 06 21:25:46.578922 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:25:46.578826    2581 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx: secret "seaweedfs-tls-serving" not found
Jul 06 21:25:46.578922 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:25:46.578893    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a8504e5-800b-429c-ba06-880a996e40ae-seaweedfs-tls-serving podName:7a8504e5-800b-429c-ba06-880a996e40ae nodeName:}" failed. No retries permitted until 2026-07-06 21:25:47.078872711 +0000 UTC m=+784.072971874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/7a8504e5-800b-429c-ba06-880a996e40ae-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-f8xfx" (UID: "7a8504e5-800b-429c-ba06-880a996e40ae") : secret "seaweedfs-tls-serving" not found
Jul 06 21:25:46.579040 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.578992    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7a8504e5-800b-429c-ba06-880a996e40ae-data\") pod \"seaweedfs-tls-serving-67b7477c6d-f8xfx\" (UID: \"7a8504e5-800b-429c-ba06-880a996e40ae\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"
Jul 06 21:25:46.586497 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:46.586467    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prhdh\" (UniqueName: \"kubernetes.io/projected/7a8504e5-800b-429c-ba06-880a996e40ae-kube-api-access-prhdh\") pod \"seaweedfs-tls-serving-67b7477c6d-f8xfx\" (UID: \"7a8504e5-800b-429c-ba06-880a996e40ae\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"
Jul 06 21:25:47.082581 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:47.082550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/7a8504e5-800b-429c-ba06-880a996e40ae-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-f8xfx\" (UID: \"7a8504e5-800b-429c-ba06-880a996e40ae\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"
Jul 06 21:25:47.084952 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:47.084931    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/7a8504e5-800b-429c-ba06-880a996e40ae-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-f8xfx\" (UID: \"7a8504e5-800b-429c-ba06-880a996e40ae\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"
Jul 06 21:25:47.328831 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:47.328794    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"
Jul 06 21:25:47.449815 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:47.449785    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx"]
Jul 06 21:25:47.452785 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:25:47.452756    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8504e5_800b_429c_ba06_880a996e40ae.slice/crio-f45a9f8400e67f84a0705a1e842aca86ad4e214be68df2d39ace5023004e7c5a WatchSource:0}: Error finding container f45a9f8400e67f84a0705a1e842aca86ad4e214be68df2d39ace5023004e7c5a: Status 404 returned error can't find the container with id f45a9f8400e67f84a0705a1e842aca86ad4e214be68df2d39ace5023004e7c5a
Jul 06 21:25:47.613377 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:47.613281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx" event={"ID":"7a8504e5-800b-429c-ba06-880a996e40ae","Type":"ContainerStarted","Data":"f45a9f8400e67f84a0705a1e842aca86ad4e214be68df2d39ace5023004e7c5a"}
Jul 06 21:25:48.618599 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:48.618504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx" event={"ID":"7a8504e5-800b-429c-ba06-880a996e40ae","Type":"ContainerStarted","Data":"0ae8d5bb01aa328d61cedf68206bfcb1836bac212f598b4e2ffdefad274708fc"}
Jul 06 21:25:48.634426 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:25:48.634380    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-f8xfx" podStartSLOduration=2.360475556 podStartE2EDuration="2.634361554s" totalImagesPullingTime="273.885998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:25:48.633007437 +0000 UTC m=+785.627106607" watchObservedRunningTime="2026-07-06 21:25:48.634361554 +0000 UTC m=+785.628460720"
Jul 06 21:26:13.960430 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:13.960349    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"]
Jul 06 21:26:13.963971 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:13.963950    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:13.966579 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:13.966550    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:26:13.966988 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:13.966810    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:26:13.966988 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:13.966853    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pnnb5\""
Jul 06 21:26:13.966988 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:13.966872    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 06 21:26:13.967181 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:13.967084    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 21:26:13.967496 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:13.967474    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 06 21:26:13.967496 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:13.967491    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:26:13.980837 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:13.980813    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"]
Jul 06 21:26:14.096750 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.096712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73277ebe-383c-4b73-be93-d5b9639ff3c1-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.096750 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.096752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73277ebe-383c-4b73-be93-d5b9639ff3c1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.096971 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.096773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73277ebe-383c-4b73-be93-d5b9639ff3c1-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.096971 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.096870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vv48\" (UniqueName: \"kubernetes.io/projected/73277ebe-383c-4b73-be93-d5b9639ff3c1-kube-api-access-7vv48\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.096971 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.096909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73277ebe-383c-4b73-be93-d5b9639ff3c1-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.198183 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.198146    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73277ebe-383c-4b73-be93-d5b9639ff3c1-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.198183 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.198185    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73277ebe-383c-4b73-be93-d5b9639ff3c1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.198417 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.198201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73277ebe-383c-4b73-be93-d5b9639ff3c1-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.198417 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.198247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vv48\" (UniqueName: \"kubernetes.io/projected/73277ebe-383c-4b73-be93-d5b9639ff3c1-kube-api-access-7vv48\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.198417 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.198372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73277ebe-383c-4b73-be93-d5b9639ff3c1-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.198579 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:26:14.198561    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 06 21:26:14.198633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.198606    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73277ebe-383c-4b73-be93-d5b9639ff3c1-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.198688 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:26:14.198613    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73277ebe-383c-4b73-be93-d5b9639ff3c1-proxy-tls podName:73277ebe-383c-4b73-be93-d5b9639ff3c1 nodeName:}" failed. No retries permitted until 2026-07-06 21:26:14.698599935 +0000 UTC m=+811.692699075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/73277ebe-383c-4b73-be93-d5b9639ff3c1-proxy-tls") pod "isvc-sklearn-batcher-predictor-956c77b89-krt5l" (UID: "73277ebe-383c-4b73-be93-d5b9639ff3c1") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 06 21:26:14.198898 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.198879    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73277ebe-383c-4b73-be93-d5b9639ff3c1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.198978 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.198954    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73277ebe-383c-4b73-be93-d5b9639ff3c1-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.207247 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.207202    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vv48\" (UniqueName: \"kubernetes.io/projected/73277ebe-383c-4b73-be93-d5b9639ff3c1-kube-api-access-7vv48\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.704291 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.704248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73277ebe-383c-4b73-be93-d5b9639ff3c1-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.706764 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.706728    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73277ebe-383c-4b73-be93-d5b9639ff3c1-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-956c77b89-krt5l\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:14.875213 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:14.875150    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:15.005772 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:15.005739    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"]
Jul 06 21:26:15.008944 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:26:15.008911    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73277ebe_383c_4b73_be93_d5b9639ff3c1.slice/crio-7faf6d471efb457dc4858acc5e2916c45ff99d237da4820eb0718037b8132285 WatchSource:0}: Error finding container 7faf6d471efb457dc4858acc5e2916c45ff99d237da4820eb0718037b8132285: Status 404 returned error can't find the container with id 7faf6d471efb457dc4858acc5e2916c45ff99d237da4820eb0718037b8132285
Jul 06 21:26:15.713828 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:15.713794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" event={"ID":"73277ebe-383c-4b73-be93-d5b9639ff3c1","Type":"ContainerStarted","Data":"7faf6d471efb457dc4858acc5e2916c45ff99d237da4820eb0718037b8132285"}
Jul 06 21:26:19.731596 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:19.731558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" event={"ID":"73277ebe-383c-4b73-be93-d5b9639ff3c1","Type":"ContainerStarted","Data":"81811adf59d3b8809af155f4ada575d9d4ad41a5f5c69970bb982100a905573e"}
Jul 06 21:26:22.743413 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:22.743329    2581 generic.go:356] "Generic (PLEG): container finished" podID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerID="81811adf59d3b8809af155f4ada575d9d4ad41a5f5c69970bb982100a905573e" exitCode=0
Jul 06 21:26:22.743747 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:22.743409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" event={"ID":"73277ebe-383c-4b73-be93-d5b9639ff3c1","Type":"ContainerDied","Data":"81811adf59d3b8809af155f4ada575d9d4ad41a5f5c69970bb982100a905573e"}
Jul 06 21:26:36.810026 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:36.809986    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" event={"ID":"73277ebe-383c-4b73-be93-d5b9639ff3c1","Type":"ContainerStarted","Data":"7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29"}
Jul 06 21:26:38.820185 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:38.820147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" event={"ID":"73277ebe-383c-4b73-be93-d5b9639ff3c1","Type":"ContainerStarted","Data":"9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645"}
Jul 06 21:26:41.833965 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:41.833922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" event={"ID":"73277ebe-383c-4b73-be93-d5b9639ff3c1","Type":"ContainerStarted","Data":"4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a"}
Jul 06 21:26:41.834430 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:41.834188    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:41.834430 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:41.834263    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:41.834430 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:41.834297    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:41.835793 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:41.835754    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 06 21:26:41.836461 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:41.836439    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:26:41.857196 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:41.857140    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podStartSLOduration=3.405787577 podStartE2EDuration="28.857121855s" totalImagesPullingTime="21.752504334s" totalInitContainerRuntime="3.698829944s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:26:13 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:26:41.855891595 +0000 UTC m=+838.849990814" watchObservedRunningTime="2026-07-06 21:26:41.857121855 +0000 UTC m=+838.851221016"
Jul 06 21:26:42.838414 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:42.838372    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 06 21:26:42.838871 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:42.838852    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:26:47.843022 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:47.842995    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:26:47.843601 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:47.843568    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 06 21:26:47.843859 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:47.843830    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:26:57.843578 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:57.843531    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 06 21:26:57.844101 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:26:57.843922    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:27:07.843550 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:27:07.843512    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 06 21:27:07.844113 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:27:07.843976    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:27:17.843472 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:27:17.843433    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 06 21:27:17.843898 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:27:17.843780    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:27:27.843630 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:27:27.843585    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 06 21:27:27.844068 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:27:27.843950    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:27:37.844423 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:27:37.844336    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 06 21:27:37.844851 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:27:37.844823    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:27:47.844209 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:27:47.844157    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 06 21:27:47.844706 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:27:47.844576    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:27:57.844010 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:27:57.843973    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:27:57.844543 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:27:57.844524    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:28:08.957981 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:08.957948    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"]
Jul 06 21:28:08.958387 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:08.958247    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" containerID="cri-o://7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29" gracePeriod=30
Jul 06 21:28:08.958387 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:08.958277    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kube-rbac-proxy" containerID="cri-o://9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645" gracePeriod=30
Jul 06 21:28:08.958387 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:08.958279    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" containerID="cri-o://4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a" gracePeriod=30
Jul 06 21:28:08.978594 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:08.978570    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"]
Jul 06 21:28:08.981988 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:08.981969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:08.984401 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:08.984378    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 06 21:28:08.984512 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:08.984492    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 06 21:28:08.994242 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:08.994201    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"]
Jul 06 21:28:09.053929 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.053894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2dhn\" (UniqueName: \"kubernetes.io/projected/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kube-api-access-p2dhn\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.054090 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.053948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.054090 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.053972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.054090 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.054041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.054090 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.054080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.142181 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.142150    2581 generic.go:356] "Generic (PLEG): container finished" podID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerID="9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645" exitCode=2
Jul 06 21:28:09.142356 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.142214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" event={"ID":"73277ebe-383c-4b73-be93-d5b9639ff3c1","Type":"ContainerDied","Data":"9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645"}
Jul 06 21:28:09.154513 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.154489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.154636 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.154517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.154636 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.154551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.154636 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.154569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.154636 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.154600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2dhn\" (UniqueName: \"kubernetes.io/projected/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kube-api-access-p2dhn\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.155046 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.155023    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.155166 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.155148    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.155265 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.155215    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.157019 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.156998    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.163033 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.163011    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2dhn\" (UniqueName: \"kubernetes.io/projected/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kube-api-access-p2dhn\") pod \"isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.293431 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.293343    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:09.424409 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:09.424383    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"]
Jul 06 21:28:09.427202 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:28:09.427175    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe42f0b6_a1a7_42ba_86ba_b1da59c4c2fe.slice/crio-062a2dbffd3ec2740f682012a39c1092fe493aa5bd00fbcea6f1d20289544149 WatchSource:0}: Error finding container 062a2dbffd3ec2740f682012a39c1092fe493aa5bd00fbcea6f1d20289544149: Status 404 returned error can't find the container with id 062a2dbffd3ec2740f682012a39c1092fe493aa5bd00fbcea6f1d20289544149
Jul 06 21:28:10.148074 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:10.148035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" event={"ID":"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe","Type":"ContainerStarted","Data":"10a9e7bcc274945e3faf2db01e81d9a7695ace4ceb0206085321b433d6e8e18a"}
Jul 06 21:28:10.148486 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:10.148121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" event={"ID":"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe","Type":"ContainerStarted","Data":"062a2dbffd3ec2740f682012a39c1092fe493aa5bd00fbcea6f1d20289544149"}
Jul 06 21:28:12.838878 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:12.838835    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 06 21:28:14.165480 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:14.165439    2581 generic.go:356] "Generic (PLEG): container finished" podID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerID="7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29" exitCode=0
Jul 06 21:28:14.165855 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:14.165512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" event={"ID":"73277ebe-383c-4b73-be93-d5b9639ff3c1","Type":"ContainerDied","Data":"7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29"}
Jul 06 21:28:14.166928 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:14.166909    2581 generic.go:356] "Generic (PLEG): container finished" podID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerID="10a9e7bcc274945e3faf2db01e81d9a7695ace4ceb0206085321b433d6e8e18a" exitCode=0
Jul 06 21:28:14.167039 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:14.166935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" event={"ID":"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe","Type":"ContainerDied","Data":"10a9e7bcc274945e3faf2db01e81d9a7695ace4ceb0206085321b433d6e8e18a"}
Jul 06 21:28:15.172458 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:15.172426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" event={"ID":"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe","Type":"ContainerStarted","Data":"8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91"}
Jul 06 21:28:15.172833 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:15.172491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" event={"ID":"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe","Type":"ContainerStarted","Data":"0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e"}
Jul 06 21:28:15.172833 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:15.172523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" event={"ID":"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe","Type":"ContainerStarted","Data":"b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6"}
Jul 06 21:28:15.172833 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:15.172822    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:15.172998 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:15.172982    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:15.174137 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:15.174114    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:28:15.193553 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:15.193507    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podStartSLOduration=3.440316351 podStartE2EDuration="7.193491501s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.75317515s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:28:15.191838501 +0000 UTC m=+932.185937665" watchObservedRunningTime="2026-07-06 21:28:15.193491501 +0000 UTC m=+932.187590661"
Jul 06 21:28:16.176504 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:16.176474    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:16.176946 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:16.176516    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:28:16.177632 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:16.177605    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:28:17.179805 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:17.179768    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:28:17.180254 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:17.180092    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:28:17.838696 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:17.838654    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 06 21:28:17.844467 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:17.844431    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:28:17.844618 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:17.844440    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 06 21:28:22.184098 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:22.184069    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:28:22.184639 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:22.184608    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:28:22.184938 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:22.184917    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:28:22.838791 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:22.838748    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 06 21:28:22.838996 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:22.838892    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:28:27.839408 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:27.839362    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 06 21:28:27.844138 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:27.844103    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:28:27.844897 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:27.844874    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 06 21:28:32.185216 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:32.185171    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:28:32.185675 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:32.185651    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:28:32.839021 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:32.838979    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 06 21:28:37.838832 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:37.838794    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 06 21:28:37.844476 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:37.844440    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 06 21:28:37.844603 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:37.844547    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:28:37.844603 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:37.844561    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:28:37.844772 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:37.844752    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:28:39.104601 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.104566    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:28:39.192945 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.192904    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73277ebe-383c-4b73-be93-d5b9639ff3c1-cabundle-cert\") pod \"73277ebe-383c-4b73-be93-d5b9639ff3c1\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") "
Jul 06 21:28:39.193114 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.192968    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73277ebe-383c-4b73-be93-d5b9639ff3c1-kube-rbac-proxy-sar-config\") pod \"73277ebe-383c-4b73-be93-d5b9639ff3c1\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") "
Jul 06 21:28:39.193114 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.192996    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73277ebe-383c-4b73-be93-d5b9639ff3c1-proxy-tls\") pod \"73277ebe-383c-4b73-be93-d5b9639ff3c1\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") "
Jul 06 21:28:39.193114 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.193030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vv48\" (UniqueName: \"kubernetes.io/projected/73277ebe-383c-4b73-be93-d5b9639ff3c1-kube-api-access-7vv48\") pod \"73277ebe-383c-4b73-be93-d5b9639ff3c1\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") "
Jul 06 21:28:39.193274 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.193196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73277ebe-383c-4b73-be93-d5b9639ff3c1-kserve-provision-location\") pod \"73277ebe-383c-4b73-be93-d5b9639ff3c1\" (UID: \"73277ebe-383c-4b73-be93-d5b9639ff3c1\") "
Jul 06 21:28:39.193378 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.193359    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73277ebe-383c-4b73-be93-d5b9639ff3c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "73277ebe-383c-4b73-be93-d5b9639ff3c1" (UID: "73277ebe-383c-4b73-be93-d5b9639ff3c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:28:39.193520 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.193489    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73277ebe-383c-4b73-be93-d5b9639ff3c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73277ebe-383c-4b73-be93-d5b9639ff3c1" (UID: "73277ebe-383c-4b73-be93-d5b9639ff3c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:28:39.193520 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.193502    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73277ebe-383c-4b73-be93-d5b9639ff3c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73277ebe-383c-4b73-be93-d5b9639ff3c1" (UID: "73277ebe-383c-4b73-be93-d5b9639ff3c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:28:39.193671 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.193543    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73277ebe-383c-4b73-be93-d5b9639ff3c1-cabundle-cert\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:28:39.195348 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.195325    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73277ebe-383c-4b73-be93-d5b9639ff3c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73277ebe-383c-4b73-be93-d5b9639ff3c1" (UID: "73277ebe-383c-4b73-be93-d5b9639ff3c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:28:39.195445 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.195328    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73277ebe-383c-4b73-be93-d5b9639ff3c1-kube-api-access-7vv48" (OuterVolumeSpecName: "kube-api-access-7vv48") pod "73277ebe-383c-4b73-be93-d5b9639ff3c1" (UID: "73277ebe-383c-4b73-be93-d5b9639ff3c1"). InnerVolumeSpecName "kube-api-access-7vv48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:28:39.264844 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.264765    2581 generic.go:356] "Generic (PLEG): container finished" podID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerID="4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a" exitCode=0
Jul 06 21:28:39.264844 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.264803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" event={"ID":"73277ebe-383c-4b73-be93-d5b9639ff3c1","Type":"ContainerDied","Data":"4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a"}
Jul 06 21:28:39.264844 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.264836    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"
Jul 06 21:28:39.265075 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.264862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l" event={"ID":"73277ebe-383c-4b73-be93-d5b9639ff3c1","Type":"ContainerDied","Data":"7faf6d471efb457dc4858acc5e2916c45ff99d237da4820eb0718037b8132285"}
Jul 06 21:28:39.265075 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.264895    2581 scope.go:117] "RemoveContainer" containerID="4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a"
Jul 06 21:28:39.272690 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.272641    2581 scope.go:117] "RemoveContainer" containerID="9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645"
Jul 06 21:28:39.279924 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.279909    2581 scope.go:117] "RemoveContainer" containerID="7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29"
Jul 06 21:28:39.285373 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.285350    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"]
Jul 06 21:28:39.286955 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.286930    2581 scope.go:117] "RemoveContainer" containerID="81811adf59d3b8809af155f4ada575d9d4ad41a5f5c69970bb982100a905573e"
Jul 06 21:28:39.289190 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.289171    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-956c77b89-krt5l"]
Jul 06 21:28:39.293872 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.293850    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vv48\" (UniqueName: \"kubernetes.io/projected/73277ebe-383c-4b73-be93-d5b9639ff3c1-kube-api-access-7vv48\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:28:39.293872 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.293869    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73277ebe-383c-4b73-be93-d5b9639ff3c1-kserve-provision-location\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:28:39.294027 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.293882    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73277ebe-383c-4b73-be93-d5b9639ff3c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:28:39.294027 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.293891    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73277ebe-383c-4b73-be93-d5b9639ff3c1-proxy-tls\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:28:39.294512 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.294496    2581 scope.go:117] "RemoveContainer" containerID="4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a"
Jul 06 21:28:39.294778 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:28:39.294761    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a\": container with ID starting with 4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a not found: ID does not exist" containerID="4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a"
Jul 06 21:28:39.294825 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.294788    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a"} err="failed to get container status \"4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a\": rpc error: code = NotFound desc = could not find container \"4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a\": container with ID starting with 4ebee64ba3ef4ab1e96212a02353b7cb05ba5cd799c75b65d3d337740de9a68a not found: ID does not exist"
Jul 06 21:28:39.294825 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.294806    2581 scope.go:117] "RemoveContainer" containerID="9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645"
Jul 06 21:28:39.295059 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:28:39.295041    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645\": container with ID starting with 9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645 not found: ID does not exist" containerID="9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645"
Jul 06 21:28:39.295107 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.295077    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645"} err="failed to get container status \"9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645\": rpc error: code = NotFound desc = could not find container \"9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645\": container with ID starting with 9d4801305df5d45e3881fec124cbb43f2cdea1e105352bcd5ae7514d0766d645 not found: ID does not exist"
Jul 06 21:28:39.295107 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.295094    2581 scope.go:117] "RemoveContainer" containerID="7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29"
Jul 06 21:28:39.295338 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:28:39.295321    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29\": container with ID starting with 7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29 not found: ID does not exist" containerID="7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29"
Jul 06 21:28:39.295396 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.295344    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29"} err="failed to get container status \"7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29\": rpc error: code = NotFound desc = could not find container \"7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29\": container with ID starting with 7b6e3a6eb4f9309dd1a64b14e7ca2f594925674f275a845885d8694589ca5f29 not found: ID does not exist"
Jul 06 21:28:39.295396 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.295359    2581 scope.go:117] "RemoveContainer" containerID="81811adf59d3b8809af155f4ada575d9d4ad41a5f5c69970bb982100a905573e"
Jul 06 21:28:39.295602 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:28:39.295585    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81811adf59d3b8809af155f4ada575d9d4ad41a5f5c69970bb982100a905573e\": container with ID starting with 81811adf59d3b8809af155f4ada575d9d4ad41a5f5c69970bb982100a905573e not found: ID does not exist" containerID="81811adf59d3b8809af155f4ada575d9d4ad41a5f5c69970bb982100a905573e"
Jul 06 21:28:39.295643 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.295608    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81811adf59d3b8809af155f4ada575d9d4ad41a5f5c69970bb982100a905573e"} err="failed to get container status \"81811adf59d3b8809af155f4ada575d9d4ad41a5f5c69970bb982100a905573e\": rpc error: code = NotFound desc = could not find container \"81811adf59d3b8809af155f4ada575d9d4ad41a5f5c69970bb982100a905573e\": container with ID starting with 81811adf59d3b8809af155f4ada575d9d4ad41a5f5c69970bb982100a905573e not found: ID does not exist"
Jul 06 21:28:39.877485 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:39.877456    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" path="/var/lib/kubelet/pods/73277ebe-383c-4b73-be93-d5b9639ff3c1/volumes"
Jul 06 21:28:42.184734 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:42.184692    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:28:42.185166 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:42.185144    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:28:52.184579 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:52.184541    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:28:52.184962 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:28:52.184880    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:29:02.184975 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:02.184934    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:29:02.185511 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:02.185466    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:29:12.185340 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:12.185244    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:29:12.185782 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:12.185756    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:29:22.184820 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:22.184782    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:29:22.185280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:22.185172    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:29:32.185301 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:32.185271    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:29:32.185816 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:32.185778    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:29:44.136105 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.136075    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"]
Jul 06 21:29:44.136625 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.136371    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" containerID="cri-o://b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6" gracePeriod=30
Jul 06 21:29:44.136625 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.136394    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" containerID="cri-o://8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91" gracePeriod=30
Jul 06 21:29:44.136625 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.136435    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kube-rbac-proxy" containerID="cri-o://0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e" gracePeriod=30
Jul 06 21:29:44.242329 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.242297    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"]
Jul 06 21:29:44.242599 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.242587    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="storage-initializer"
Jul 06 21:29:44.242645 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.242601    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="storage-initializer"
Jul 06 21:29:44.242645 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.242615    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container"
Jul 06 21:29:44.242645 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.242621    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container"
Jul 06 21:29:44.242645 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.242637    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kube-rbac-proxy"
Jul 06 21:29:44.242645 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.242643    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kube-rbac-proxy"
Jul 06 21:29:44.242819 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.242653    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent"
Jul 06 21:29:44.242819 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.242658    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent"
Jul 06 21:29:44.242819 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.242704    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kserve-container"
Jul 06 21:29:44.242819 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.242715    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="kube-rbac-proxy"
Jul 06 21:29:44.242819 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.242721    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="73277ebe-383c-4b73-be93-d5b9639ff3c1" containerName="agent"
Jul 06 21:29:44.245722 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.245701    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:44.248187 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.248153    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 06 21:29:44.248346 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.248187    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 06 21:29:44.255561 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.255539    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"]
Jul 06 21:29:44.383030 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.382999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3221ede-d447-45e5-acba-8bab4b29b92f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-wwslp\" (UID: \"b3221ede-d447-45e5-acba-8bab4b29b92f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:44.383185 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.383061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3221ede-d447-45e5-acba-8bab4b29b92f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-wwslp\" (UID: \"b3221ede-d447-45e5-acba-8bab4b29b92f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:44.383185 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.383086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj2j9\" (UniqueName: \"kubernetes.io/projected/b3221ede-d447-45e5-acba-8bab4b29b92f-kube-api-access-bj2j9\") pod \"message-dumper-predictor-6575bcfdfd-wwslp\" (UID: \"b3221ede-d447-45e5-acba-8bab4b29b92f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:44.484337 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.484301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3221ede-d447-45e5-acba-8bab4b29b92f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-wwslp\" (UID: \"b3221ede-d447-45e5-acba-8bab4b29b92f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:44.484512 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.484343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj2j9\" (UniqueName: \"kubernetes.io/projected/b3221ede-d447-45e5-acba-8bab4b29b92f-kube-api-access-bj2j9\") pod \"message-dumper-predictor-6575bcfdfd-wwslp\" (UID: \"b3221ede-d447-45e5-acba-8bab4b29b92f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:44.484512 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.484386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3221ede-d447-45e5-acba-8bab4b29b92f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-wwslp\" (UID: \"b3221ede-d447-45e5-acba-8bab4b29b92f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:44.485018 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.484997    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3221ede-d447-45e5-acba-8bab4b29b92f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-wwslp\" (UID: \"b3221ede-d447-45e5-acba-8bab4b29b92f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:44.486785 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.486760    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3221ede-d447-45e5-acba-8bab4b29b92f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-wwslp\" (UID: \"b3221ede-d447-45e5-acba-8bab4b29b92f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:44.492932 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.492904    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj2j9\" (UniqueName: \"kubernetes.io/projected/b3221ede-d447-45e5-acba-8bab4b29b92f-kube-api-access-bj2j9\") pod \"message-dumper-predictor-6575bcfdfd-wwslp\" (UID: \"b3221ede-d447-45e5-acba-8bab4b29b92f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:44.494766 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.494740    2581 generic.go:356] "Generic (PLEG): container finished" podID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerID="0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e" exitCode=2
Jul 06 21:29:44.494857 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.494802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" event={"ID":"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe","Type":"ContainerDied","Data":"0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e"}
Jul 06 21:29:44.557728 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.557688    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:44.687275 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.687106    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"]
Jul 06 21:29:44.689791 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:29:44.689762    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3221ede_d447_45e5_acba_8bab4b29b92f.slice/crio-b8e67d7386cdd63ce1c0e2913bc5bc6d3b3362d7ed64195ba78a35dd4ff78a95 WatchSource:0}: Error finding container b8e67d7386cdd63ce1c0e2913bc5bc6d3b3362d7ed64195ba78a35dd4ff78a95: Status 404 returned error can't find the container with id b8e67d7386cdd63ce1c0e2913bc5bc6d3b3362d7ed64195ba78a35dd4ff78a95
Jul 06 21:29:44.691573 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:44.691560    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:29:45.499627 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:45.499597    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp" event={"ID":"b3221ede-d447-45e5-acba-8bab4b29b92f","Type":"ContainerStarted","Data":"b8e67d7386cdd63ce1c0e2913bc5bc6d3b3362d7ed64195ba78a35dd4ff78a95"}
Jul 06 21:29:46.504474 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:46.504436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp" event={"ID":"b3221ede-d447-45e5-acba-8bab4b29b92f","Type":"ContainerStarted","Data":"e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed"}
Jul 06 21:29:46.504853 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:46.504493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp" event={"ID":"b3221ede-d447-45e5-acba-8bab4b29b92f","Type":"ContainerStarted","Data":"789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc"}
Jul 06 21:29:46.504853 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:46.504552    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:46.522870 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:46.522824    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp" podStartSLOduration=1.280666026 podStartE2EDuration="2.522808628s" totalImagesPullingTime="1.242142602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:29:46.521777816 +0000 UTC m=+1023.515876989" watchObservedRunningTime="2026-07-06 21:29:46.522808628 +0000 UTC m=+1023.516907788"
Jul 06 21:29:47.180500 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:47.180458    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 06 21:29:47.508859 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:47.508766    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:47.510396 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:47.510377    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:49.518109 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:49.518080    2581 generic.go:356] "Generic (PLEG): container finished" podID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerID="b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6" exitCode=0
Jul 06 21:29:49.518616 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:49.518162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" event={"ID":"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe","Type":"ContainerDied","Data":"b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6"}
Jul 06 21:29:52.180048 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:52.180008    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 06 21:29:52.185779 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:52.185740    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:29:52.185916 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:52.185810    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:29:54.523261 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:54.523219    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:29:57.180211 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:57.180168    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 06 21:29:57.180656 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:29:57.180378    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:30:02.180073 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:02.180025    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 06 21:30:02.185710 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:02.185685    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:30:02.185828 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:02.185750    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:30:04.337894 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.337861    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"]
Jul 06 21:30:04.341806 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.341788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.345177 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.345153    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 06 21:30:04.345840 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.345823    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 06 21:30:04.368242 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.368205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"]
Jul 06 21:30:04.434544 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.434505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74bad355-8e2d-4276-b23a-d713fcca540e-cabundle-cert\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.434544 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.434544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74bad355-8e2d-4276-b23a-d713fcca540e-proxy-tls\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.434760 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.434621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74bad355-8e2d-4276-b23a-d713fcca540e-kserve-provision-location\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.434760 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.434663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk5z2\" (UniqueName: \"kubernetes.io/projected/74bad355-8e2d-4276-b23a-d713fcca540e-kube-api-access-fk5z2\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.434760 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.434682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74bad355-8e2d-4276-b23a-d713fcca540e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.535531 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.535499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk5z2\" (UniqueName: \"kubernetes.io/projected/74bad355-8e2d-4276-b23a-d713fcca540e-kube-api-access-fk5z2\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.535531 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.535533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74bad355-8e2d-4276-b23a-d713fcca540e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.535799 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.535567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74bad355-8e2d-4276-b23a-d713fcca540e-cabundle-cert\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.535799 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.535587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74bad355-8e2d-4276-b23a-d713fcca540e-proxy-tls\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.535799 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.535611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74bad355-8e2d-4276-b23a-d713fcca540e-kserve-provision-location\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.535799 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:30:04.535723    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 06 21:30:04.535799 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:30:04.535782    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74bad355-8e2d-4276-b23a-d713fcca540e-proxy-tls podName:74bad355-8e2d-4276-b23a-d713fcca540e nodeName:}" failed. No retries permitted until 2026-07-06 21:30:05.035761478 +0000 UTC m=+1042.029860633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/74bad355-8e2d-4276-b23a-d713fcca540e-proxy-tls") pod "isvc-logger-predictor-757779fc66-s4dtm" (UID: "74bad355-8e2d-4276-b23a-d713fcca540e") : secret "isvc-logger-predictor-serving-cert" not found
Jul 06 21:30:04.536072 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.536054    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74bad355-8e2d-4276-b23a-d713fcca540e-kserve-provision-location\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.536309 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.536284    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74bad355-8e2d-4276-b23a-d713fcca540e-cabundle-cert\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.536394 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.536337    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74bad355-8e2d-4276-b23a-d713fcca540e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:04.546071 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:04.546047    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk5z2\" (UniqueName: \"kubernetes.io/projected/74bad355-8e2d-4276-b23a-d713fcca540e-kube-api-access-fk5z2\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:05.038426 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:05.038387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74bad355-8e2d-4276-b23a-d713fcca540e-proxy-tls\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:05.040852 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:05.040819    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74bad355-8e2d-4276-b23a-d713fcca540e-proxy-tls\") pod \"isvc-logger-predictor-757779fc66-s4dtm\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:05.252072 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:05.252029    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:05.384780 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:05.384751    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"]
Jul 06 21:30:05.389131 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:30:05.389105    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bad355_8e2d_4276_b23a_d713fcca540e.slice/crio-cd082ba777bd60358522902372e68f8c0b96137b87393ad73cd36f74fe10dfbd WatchSource:0}: Error finding container cd082ba777bd60358522902372e68f8c0b96137b87393ad73cd36f74fe10dfbd: Status 404 returned error can't find the container with id cd082ba777bd60358522902372e68f8c0b96137b87393ad73cd36f74fe10dfbd
Jul 06 21:30:05.578829 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:05.578737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" event={"ID":"74bad355-8e2d-4276-b23a-d713fcca540e","Type":"ContainerStarted","Data":"b6e307e08d883f07aed1537d0f3186a6f9dd5191924b961c56d4c01ca6b907aa"}
Jul 06 21:30:05.578990 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:05.578826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" event={"ID":"74bad355-8e2d-4276-b23a-d713fcca540e","Type":"ContainerStarted","Data":"cd082ba777bd60358522902372e68f8c0b96137b87393ad73cd36f74fe10dfbd"}
Jul 06 21:30:07.180273 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:07.180212    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 06 21:30:09.594951 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:09.594870    2581 generic.go:356] "Generic (PLEG): container finished" podID="74bad355-8e2d-4276-b23a-d713fcca540e" containerID="b6e307e08d883f07aed1537d0f3186a6f9dd5191924b961c56d4c01ca6b907aa" exitCode=0
Jul 06 21:30:09.595326 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:09.594946    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" event={"ID":"74bad355-8e2d-4276-b23a-d713fcca540e","Type":"ContainerDied","Data":"b6e307e08d883f07aed1537d0f3186a6f9dd5191924b961c56d4c01ca6b907aa"}
Jul 06 21:30:10.601247 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:10.601200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" event={"ID":"74bad355-8e2d-4276-b23a-d713fcca540e","Type":"ContainerStarted","Data":"51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31"}
Jul 06 21:30:10.601625 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:10.601279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" event={"ID":"74bad355-8e2d-4276-b23a-d713fcca540e","Type":"ContainerStarted","Data":"b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce"}
Jul 06 21:30:10.601625 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:10.601303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" event={"ID":"74bad355-8e2d-4276-b23a-d713fcca540e","Type":"ContainerStarted","Data":"79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c"}
Jul 06 21:30:10.601625 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:10.601578    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:10.601732 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:10.601640    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:10.601732 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:10.601679    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:10.602961 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:10.602932    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:30:10.603585 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:10.603557    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:30:10.624499 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:10.624451    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podStartSLOduration=2.770409237 podStartE2EDuration="6.624436394s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.854027157s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:30:10.623004634 +0000 UTC m=+1047.617103820" watchObservedRunningTime="2026-07-06 21:30:10.624436394 +0000 UTC m=+1047.618535554"
Jul 06 21:30:11.605670 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:11.605622    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:30:11.606244 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:11.606205    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:30:12.180087 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:12.180039    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 06 21:30:12.185721 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:12.185686    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 06 21:30:12.185841 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:12.185780    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:30:12.185841 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:12.185820    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:30:12.185946 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:12.185935    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:30:14.282788 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.282763    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:30:14.311838 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.311813    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kserve-provision-location\") pod \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") "
Jul 06 21:30:14.312013 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.311872    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kube-rbac-proxy-sar-config\") pod \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") "
Jul 06 21:30:14.312013 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.311902    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2dhn\" (UniqueName: \"kubernetes.io/projected/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kube-api-access-p2dhn\") pod \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") "
Jul 06 21:30:14.312013 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.311939    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-proxy-tls\") pod \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") "
Jul 06 21:30:14.312013 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.311967    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-cabundle-cert\") pod \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\" (UID: \"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe\") "
Jul 06 21:30:14.312222 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.312147    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" (UID: "fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:30:14.312382 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.312219    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" (UID: "fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:30:14.312504 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.312411    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" (UID: "fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:30:14.313954 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.313930    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kube-api-access-p2dhn" (OuterVolumeSpecName: "kube-api-access-p2dhn") pod "fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" (UID: "fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe"). InnerVolumeSpecName "kube-api-access-p2dhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:30:14.314024 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.313980    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" (UID: "fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:30:14.413585 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.413548    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:30:14.413723 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.413588    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2dhn\" (UniqueName: \"kubernetes.io/projected/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kube-api-access-p2dhn\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:30:14.413723 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.413604    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-proxy-tls\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:30:14.413723 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.413618    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-cabundle-cert\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:30:14.413723 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.413632    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe-kserve-provision-location\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:30:14.620564 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.620535    2581 generic.go:356] "Generic (PLEG): container finished" podID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerID="8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91" exitCode=0
Jul 06 21:30:14.620731 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.620600    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"
Jul 06 21:30:14.620731 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.620601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" event={"ID":"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe","Type":"ContainerDied","Data":"8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91"}
Jul 06 21:30:14.620731 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.620671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv" event={"ID":"fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe","Type":"ContainerDied","Data":"062a2dbffd3ec2740f682012a39c1092fe493aa5bd00fbcea6f1d20289544149"}
Jul 06 21:30:14.620731 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.620704    2581 scope.go:117] "RemoveContainer" containerID="8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91"
Jul 06 21:30:14.628893 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.628846    2581 scope.go:117] "RemoveContainer" containerID="0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e"
Jul 06 21:30:14.636162 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.636145    2581 scope.go:117] "RemoveContainer" containerID="b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6"
Jul 06 21:30:14.641723 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.641694    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"]
Jul 06 21:30:14.644117 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.644100    2581 scope.go:117] "RemoveContainer" containerID="10a9e7bcc274945e3faf2db01e81d9a7695ace4ceb0206085321b433d6e8e18a"
Jul 06 21:30:14.645036 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.645013    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-684888dfc6-gwbxv"]
Jul 06 21:30:14.651079 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.651063    2581 scope.go:117] "RemoveContainer" containerID="8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91"
Jul 06 21:30:14.651332 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:30:14.651315    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91\": container with ID starting with 8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91 not found: ID does not exist" containerID="8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91"
Jul 06 21:30:14.651380 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.651341    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91"} err="failed to get container status \"8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91\": rpc error: code = NotFound desc = could not find container \"8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91\": container with ID starting with 8579eb45b8a7bbd15adde97e3bdbae19e9f56dbaa554207ba8e4ff4d3a81dd91 not found: ID does not exist"
Jul 06 21:30:14.651380 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.651358    2581 scope.go:117] "RemoveContainer" containerID="0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e"
Jul 06 21:30:14.651572 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:30:14.651557    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e\": container with ID starting with 0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e not found: ID does not exist" containerID="0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e"
Jul 06 21:30:14.651616 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.651576    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e"} err="failed to get container status \"0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e\": rpc error: code = NotFound desc = could not find container \"0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e\": container with ID starting with 0844f30e72e1b2c7ec4af6d9be2cdc49c9b255f84bb9f45dc182c5ffa94d4a7e not found: ID does not exist"
Jul 06 21:30:14.651616 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.651589    2581 scope.go:117] "RemoveContainer" containerID="b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6"
Jul 06 21:30:14.651824 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:30:14.651806    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6\": container with ID starting with b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6 not found: ID does not exist" containerID="b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6"
Jul 06 21:30:14.651863 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.651830    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6"} err="failed to get container status \"b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6\": rpc error: code = NotFound desc = could not find container \"b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6\": container with ID starting with b570d696277d9ba36bcd296c7f02e57408174c09f796b43cf23b11dc2c8098a6 not found: ID does not exist"
Jul 06 21:30:14.651863 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.651845    2581 scope.go:117] "RemoveContainer" containerID="10a9e7bcc274945e3faf2db01e81d9a7695ace4ceb0206085321b433d6e8e18a"
Jul 06 21:30:14.652069 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:30:14.652054    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a9e7bcc274945e3faf2db01e81d9a7695ace4ceb0206085321b433d6e8e18a\": container with ID starting with 10a9e7bcc274945e3faf2db01e81d9a7695ace4ceb0206085321b433d6e8e18a not found: ID does not exist" containerID="10a9e7bcc274945e3faf2db01e81d9a7695ace4ceb0206085321b433d6e8e18a"
Jul 06 21:30:14.652108 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:14.652072    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a9e7bcc274945e3faf2db01e81d9a7695ace4ceb0206085321b433d6e8e18a"} err="failed to get container status \"10a9e7bcc274945e3faf2db01e81d9a7695ace4ceb0206085321b433d6e8e18a\": rpc error: code = NotFound desc = could not find container \"10a9e7bcc274945e3faf2db01e81d9a7695ace4ceb0206085321b433d6e8e18a\": container with ID starting with 10a9e7bcc274945e3faf2db01e81d9a7695ace4ceb0206085321b433d6e8e18a not found: ID does not exist"
Jul 06 21:30:15.878730 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:15.878697    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" path="/var/lib/kubelet/pods/fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe/volumes"
Jul 06 21:30:16.610403 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:16.610377    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:30:16.611024 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:16.610986    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:30:16.611263 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:16.611219    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:30:26.611107 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:26.611066    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:30:26.611572 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:26.611532    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:30:36.611914 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:36.611822    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:30:36.612340 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:36.612320    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:30:46.611062 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:46.611019    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:30:46.611545 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:46.611520    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:30:56.611277 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:56.611220    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:30:56.611739 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:30:56.611715    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:31:06.611270 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:06.611211    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:31:06.611727 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:06.611652    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:31:16.611761 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:16.611722    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:31:16.612265 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:16.612218    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:31:21.873946 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:21.873886    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:31:21.874416 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:21.874285    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:31:31.877818 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:31.877789    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:31:31.878289 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:31.877867    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:31:39.282362 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.282334    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-wwslp_b3221ede-d447-45e5-acba-8bab4b29b92f/kserve-container/0.log"
Jul 06 21:31:39.554372 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.554285    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"]
Jul 06 21:31:39.554845 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.554810    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" containerID="cri-o://79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c" gracePeriod=30
Jul 06 21:31:39.555027 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.555000    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" containerID="cri-o://51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31" gracePeriod=30
Jul 06 21:31:39.555124 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.555106    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kube-rbac-proxy" containerID="cri-o://b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce" gracePeriod=30
Jul 06 21:31:39.581222 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.581195    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"]
Jul 06 21:31:39.581585 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.581571    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent"
Jul 06 21:31:39.581632 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.581587    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent"
Jul 06 21:31:39.581632 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.581597    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container"
Jul 06 21:31:39.581632 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.581603    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container"
Jul 06 21:31:39.581632 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.581611    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="storage-initializer"
Jul 06 21:31:39.581632 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.581616    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="storage-initializer"
Jul 06 21:31:39.581632 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.581626    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kube-rbac-proxy"
Jul 06 21:31:39.581817 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.581635    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kube-rbac-proxy"
Jul 06 21:31:39.581817 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.581681    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kserve-container"
Jul 06 21:31:39.581817 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.581687    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="kube-rbac-proxy"
Jul 06 21:31:39.581817 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.581693    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe42f0b6-a1a7-42ba-86ba-b1da59c4c2fe" containerName="agent"
Jul 06 21:31:39.585144 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.585124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.587402 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.587376    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 06 21:31:39.587537 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.587511    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 06 21:31:39.595151 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.595130    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"]
Jul 06 21:31:39.595373 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.595338    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp" podUID="b3221ede-d447-45e5-acba-8bab4b29b92f" containerName="kserve-container" containerID="cri-o://789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc" gracePeriod=30
Jul 06 21:31:39.595444 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.595414    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp" podUID="b3221ede-d447-45e5-acba-8bab4b29b92f" containerName="kube-rbac-proxy" containerID="cri-o://e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed" gracePeriod=30
Jul 06 21:31:39.599504 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.599348    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"]
Jul 06 21:31:39.642768 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.642740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8379cd1e-92b2-47d2-9e9d-cc6861e44030-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.642914 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.642777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.642914 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.642793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vq6k\" (UniqueName: \"kubernetes.io/projected/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kube-api-access-8vq6k\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.642914 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.642832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8379cd1e-92b2-47d2-9e9d-cc6861e44030-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.642914 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.642883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.744471 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.743959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.744471 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.744064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8379cd1e-92b2-47d2-9e9d-cc6861e44030-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.744471 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.744095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.744471 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.744122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vq6k\" (UniqueName: \"kubernetes.io/projected/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kube-api-access-8vq6k\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.744471 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.744153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8379cd1e-92b2-47d2-9e9d-cc6861e44030-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.745003 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.744952    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8379cd1e-92b2-47d2-9e9d-cc6861e44030-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.745146 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.745129    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.745330 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.745311    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.747738 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.747715    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8379cd1e-92b2-47d2-9e9d-cc6861e44030-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.751925 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.751905    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vq6k\" (UniqueName: \"kubernetes.io/projected/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kube-api-access-8vq6k\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.838355 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.838331    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:31:39.844770 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.844752    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3221ede-d447-45e5-acba-8bab4b29b92f-kube-rbac-proxy-sar-config\") pod \"b3221ede-d447-45e5-acba-8bab4b29b92f\" (UID: \"b3221ede-d447-45e5-acba-8bab4b29b92f\") "
Jul 06 21:31:39.844851 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.844794    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3221ede-d447-45e5-acba-8bab4b29b92f-proxy-tls\") pod \"b3221ede-d447-45e5-acba-8bab4b29b92f\" (UID: \"b3221ede-d447-45e5-acba-8bab4b29b92f\") "
Jul 06 21:31:39.844851 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.844819    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj2j9\" (UniqueName: \"kubernetes.io/projected/b3221ede-d447-45e5-acba-8bab4b29b92f-kube-api-access-bj2j9\") pod \"b3221ede-d447-45e5-acba-8bab4b29b92f\" (UID: \"b3221ede-d447-45e5-acba-8bab4b29b92f\") "
Jul 06 21:31:39.845095 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.845071    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3221ede-d447-45e5-acba-8bab4b29b92f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b3221ede-d447-45e5-acba-8bab4b29b92f" (UID: "b3221ede-d447-45e5-acba-8bab4b29b92f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:31:39.846667 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.846640    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3221ede-d447-45e5-acba-8bab4b29b92f-kube-api-access-bj2j9" (OuterVolumeSpecName: "kube-api-access-bj2j9") pod "b3221ede-d447-45e5-acba-8bab4b29b92f" (UID: "b3221ede-d447-45e5-acba-8bab4b29b92f"). InnerVolumeSpecName "kube-api-access-bj2j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:31:39.846772 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.846709    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3221ede-d447-45e5-acba-8bab4b29b92f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3221ede-d447-45e5-acba-8bab4b29b92f" (UID: "b3221ede-d447-45e5-acba-8bab4b29b92f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:31:39.896023 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.895994    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:31:39.941138 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.941107    2581 generic.go:356] "Generic (PLEG): container finished" podID="b3221ede-d447-45e5-acba-8bab4b29b92f" containerID="e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed" exitCode=2
Jul 06 21:31:39.941138 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.941134    2581 generic.go:356] "Generic (PLEG): container finished" podID="b3221ede-d447-45e5-acba-8bab4b29b92f" containerID="789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc" exitCode=2
Jul 06 21:31:39.941408 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.941174    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"
Jul 06 21:31:39.941408 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.941184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp" event={"ID":"b3221ede-d447-45e5-acba-8bab4b29b92f","Type":"ContainerDied","Data":"e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed"}
Jul 06 21:31:39.941408 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.941262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp" event={"ID":"b3221ede-d447-45e5-acba-8bab4b29b92f","Type":"ContainerDied","Data":"789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc"}
Jul 06 21:31:39.941408 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.941296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp" event={"ID":"b3221ede-d447-45e5-acba-8bab4b29b92f","Type":"ContainerDied","Data":"b8e67d7386cdd63ce1c0e2913bc5bc6d3b3362d7ed64195ba78a35dd4ff78a95"}
Jul 06 21:31:39.941408 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.941325    2581 scope.go:117] "RemoveContainer" containerID="e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed"
Jul 06 21:31:39.946260 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.945214    2581 generic.go:356] "Generic (PLEG): container finished" podID="74bad355-8e2d-4276-b23a-d713fcca540e" containerID="b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce" exitCode=2
Jul 06 21:31:39.946260 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.945326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" event={"ID":"74bad355-8e2d-4276-b23a-d713fcca540e","Type":"ContainerDied","Data":"b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce"}
Jul 06 21:31:39.946260 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.945698    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3221ede-d447-45e5-acba-8bab4b29b92f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:31:39.946260 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.945718    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3221ede-d447-45e5-acba-8bab4b29b92f-proxy-tls\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:31:39.946260 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.945728    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj2j9\" (UniqueName: \"kubernetes.io/projected/b3221ede-d447-45e5-acba-8bab4b29b92f-kube-api-access-bj2j9\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:31:39.951209 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.951190    2581 scope.go:117] "RemoveContainer" containerID="789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc"
Jul 06 21:31:39.959069 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.958671    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"]
Jul 06 21:31:39.961024 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.960993    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wwslp"]
Jul 06 21:31:39.963202 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.963177    2581 scope.go:117] "RemoveContainer" containerID="e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed"
Jul 06 21:31:39.963609 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:31:39.963582    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed\": container with ID starting with e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed not found: ID does not exist" containerID="e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed"
Jul 06 21:31:39.963684 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.963622    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed"} err="failed to get container status \"e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed\": rpc error: code = NotFound desc = could not find container \"e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed\": container with ID starting with e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed not found: ID does not exist"
Jul 06 21:31:39.963684 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.963662    2581 scope.go:117] "RemoveContainer" containerID="789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc"
Jul 06 21:31:39.963973 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:31:39.963952    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc\": container with ID starting with 789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc not found: ID does not exist" containerID="789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc"
Jul 06 21:31:39.964044 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.963981    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc"} err="failed to get container status \"789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc\": rpc error: code = NotFound desc = could not find container \"789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc\": container with ID starting with 789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc not found: ID does not exist"
Jul 06 21:31:39.964044 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.964003    2581 scope.go:117] "RemoveContainer" containerID="e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed"
Jul 06 21:31:39.964284 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.964221    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed"} err="failed to get container status \"e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed\": rpc error: code = NotFound desc = could not find container \"e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed\": container with ID starting with e1562117059cc93278d4a633f8bd1914a2ca842ac0439479d6941e386d324fed not found: ID does not exist"
Jul 06 21:31:39.964284 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.964272    2581 scope.go:117] "RemoveContainer" containerID="789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc"
Jul 06 21:31:39.964561 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:39.964540    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc"} err="failed to get container status \"789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc\": rpc error: code = NotFound desc = could not find container \"789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc\": container with ID starting with 789117089cff27ff25ce8a1960455e7acf1cff0ebd9326fb3e18eee29e6d69cc not found: ID does not exist"
Jul 06 21:31:40.026895 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:40.026874    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"]
Jul 06 21:31:40.029799 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:31:40.029774    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8379cd1e_92b2_47d2_9e9d_cc6861e44030.slice/crio-cfc9bc6f585a66322d6667ae1a1538be54c26a5df55aba54693a233a90a17029 WatchSource:0}: Error finding container cfc9bc6f585a66322d6667ae1a1538be54c26a5df55aba54693a233a90a17029: Status 404 returned error can't find the container with id cfc9bc6f585a66322d6667ae1a1538be54c26a5df55aba54693a233a90a17029
Jul 06 21:31:40.951322 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:40.951289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" event={"ID":"8379cd1e-92b2-47d2-9e9d-cc6861e44030","Type":"ContainerStarted","Data":"fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e"}
Jul 06 21:31:40.951693 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:40.951351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" event={"ID":"8379cd1e-92b2-47d2-9e9d-cc6861e44030","Type":"ContainerStarted","Data":"cfc9bc6f585a66322d6667ae1a1538be54c26a5df55aba54693a233a90a17029"}
Jul 06 21:31:41.606800 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:41.606754    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 06 21:31:41.874002 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:41.873894    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:31:41.874271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:41.874216    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:31:41.877346 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:41.877325    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3221ede-d447-45e5-acba-8bab4b29b92f" path="/var/lib/kubelet/pods/b3221ede-d447-45e5-acba-8bab4b29b92f/volumes"
Jul 06 21:31:44.968368 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:44.968331    2581 generic.go:356] "Generic (PLEG): container finished" podID="74bad355-8e2d-4276-b23a-d713fcca540e" containerID="79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c" exitCode=0
Jul 06 21:31:44.968795 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:44.968391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" event={"ID":"74bad355-8e2d-4276-b23a-d713fcca540e","Type":"ContainerDied","Data":"79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c"}
Jul 06 21:31:46.606409 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:46.606363    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 06 21:31:51.605860 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:51.605814    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 06 21:31:51.606317 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:51.605963    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:31:51.874301 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:51.874188    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:31:51.874565 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:51.874538    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:31:56.606727 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:31:56.606693    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 06 21:32:01.606614 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:01.606563    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 06 21:32:01.874096 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:01.873993    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 06 21:32:01.874435 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:01.874406    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 21:32:01.877244 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:01.877211    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:32:01.877407 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:01.877337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:32:06.606911 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:06.606822    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 06 21:32:09.709305 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.709279    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:32:09.762419 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.762386    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74bad355-8e2d-4276-b23a-d713fcca540e-cabundle-cert\") pod \"74bad355-8e2d-4276-b23a-d713fcca540e\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") "
Jul 06 21:32:09.762605 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.762431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74bad355-8e2d-4276-b23a-d713fcca540e-proxy-tls\") pod \"74bad355-8e2d-4276-b23a-d713fcca540e\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") "
Jul 06 21:32:09.762605 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.762465    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74bad355-8e2d-4276-b23a-d713fcca540e-kserve-provision-location\") pod \"74bad355-8e2d-4276-b23a-d713fcca540e\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") "
Jul 06 21:32:09.762605 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.762489    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74bad355-8e2d-4276-b23a-d713fcca540e-kube-rbac-proxy-sar-config\") pod \"74bad355-8e2d-4276-b23a-d713fcca540e\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") "
Jul 06 21:32:09.762605 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.762531    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk5z2\" (UniqueName: \"kubernetes.io/projected/74bad355-8e2d-4276-b23a-d713fcca540e-kube-api-access-fk5z2\") pod \"74bad355-8e2d-4276-b23a-d713fcca540e\" (UID: \"74bad355-8e2d-4276-b23a-d713fcca540e\") "
Jul 06 21:32:09.762836 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.762803    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bad355-8e2d-4276-b23a-d713fcca540e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74bad355-8e2d-4276-b23a-d713fcca540e" (UID: "74bad355-8e2d-4276-b23a-d713fcca540e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:32:09.762887 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.762834    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bad355-8e2d-4276-b23a-d713fcca540e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74bad355-8e2d-4276-b23a-d713fcca540e" (UID: "74bad355-8e2d-4276-b23a-d713fcca540e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:32:09.762887 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.762842    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bad355-8e2d-4276-b23a-d713fcca540e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74bad355-8e2d-4276-b23a-d713fcca540e" (UID: "74bad355-8e2d-4276-b23a-d713fcca540e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:32:09.764636 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.764613    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74bad355-8e2d-4276-b23a-d713fcca540e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74bad355-8e2d-4276-b23a-d713fcca540e" (UID: "74bad355-8e2d-4276-b23a-d713fcca540e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:32:09.764764 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.764617    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bad355-8e2d-4276-b23a-d713fcca540e-kube-api-access-fk5z2" (OuterVolumeSpecName: "kube-api-access-fk5z2") pod "74bad355-8e2d-4276-b23a-d713fcca540e" (UID: "74bad355-8e2d-4276-b23a-d713fcca540e"). InnerVolumeSpecName "kube-api-access-fk5z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:32:09.863639 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.863547    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk5z2\" (UniqueName: \"kubernetes.io/projected/74bad355-8e2d-4276-b23a-d713fcca540e-kube-api-access-fk5z2\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:32:09.863639 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.863584    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74bad355-8e2d-4276-b23a-d713fcca540e-cabundle-cert\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:32:09.863639 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.863594    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74bad355-8e2d-4276-b23a-d713fcca540e-proxy-tls\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:32:09.863639 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.863602    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74bad355-8e2d-4276-b23a-d713fcca540e-kserve-provision-location\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:32:09.863639 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:09.863612    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74bad355-8e2d-4276-b23a-d713fcca540e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:32:10.060376 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.060341    2581 generic.go:356] "Generic (PLEG): container finished" podID="74bad355-8e2d-4276-b23a-d713fcca540e" containerID="51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31" exitCode=137
Jul 06 21:32:10.060540 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.060403    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"
Jul 06 21:32:10.060540 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.060418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" event={"ID":"74bad355-8e2d-4276-b23a-d713fcca540e","Type":"ContainerDied","Data":"51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31"}
Jul 06 21:32:10.060540 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.060490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm" event={"ID":"74bad355-8e2d-4276-b23a-d713fcca540e","Type":"ContainerDied","Data":"cd082ba777bd60358522902372e68f8c0b96137b87393ad73cd36f74fe10dfbd"}
Jul 06 21:32:10.060540 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.060535    2581 scope.go:117] "RemoveContainer" containerID="51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31"
Jul 06 21:32:10.061912 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.061885    2581 generic.go:356] "Generic (PLEG): container finished" podID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerID="fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e" exitCode=0
Jul 06 21:32:10.062031 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.061931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" event={"ID":"8379cd1e-92b2-47d2-9e9d-cc6861e44030","Type":"ContainerDied","Data":"fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e"}
Jul 06 21:32:10.073774 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.073752    2581 scope.go:117] "RemoveContainer" containerID="b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce"
Jul 06 21:32:10.079434 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.079415    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"]
Jul 06 21:32:10.081841 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.081821    2581 scope.go:117] "RemoveContainer" containerID="79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c"
Jul 06 21:32:10.084890 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.084866    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-757779fc66-s4dtm"]
Jul 06 21:32:10.089292 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.089276    2581 scope.go:117] "RemoveContainer" containerID="b6e307e08d883f07aed1537d0f3186a6f9dd5191924b961c56d4c01ca6b907aa"
Jul 06 21:32:10.095870 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.095856    2581 scope.go:117] "RemoveContainer" containerID="51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31"
Jul 06 21:32:10.096103 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:32:10.096085    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31\": container with ID starting with 51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31 not found: ID does not exist" containerID="51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31"
Jul 06 21:32:10.096170 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.096115    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31"} err="failed to get container status \"51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31\": rpc error: code = NotFound desc = could not find container \"51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31\": container with ID starting with 51a250da784f03250a058c353dd9347f690636cdc7d464587857edd3de904f31 not found: ID does not exist"
Jul 06 21:32:10.096170 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.096139    2581 scope.go:117] "RemoveContainer" containerID="b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce"
Jul 06 21:32:10.096416 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:32:10.096400    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce\": container with ID starting with b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce not found: ID does not exist" containerID="b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce"
Jul 06 21:32:10.096457 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.096423    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce"} err="failed to get container status \"b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce\": rpc error: code = NotFound desc = could not find container \"b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce\": container with ID starting with b036370abb3fc4148877419670f06d3fe818f5175cacac8c0b3c3ba1358a50ce not found: ID does not exist"
Jul 06 21:32:10.096457 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.096438    2581 scope.go:117] "RemoveContainer" containerID="79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c"
Jul 06 21:32:10.096669 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:32:10.096654    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c\": container with ID starting with 79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c not found: ID does not exist" containerID="79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c"
Jul 06 21:32:10.096704 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.096676    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c"} err="failed to get container status \"79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c\": rpc error: code = NotFound desc = could not find container \"79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c\": container with ID starting with 79f372492362f991c0ce877b5154fac42d1ff21dd9f1a97f2f65cfa55b9dd33c not found: ID does not exist"
Jul 06 21:32:10.096704 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.096690    2581 scope.go:117] "RemoveContainer" containerID="b6e307e08d883f07aed1537d0f3186a6f9dd5191924b961c56d4c01ca6b907aa"
Jul 06 21:32:10.096894 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:32:10.096873    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e307e08d883f07aed1537d0f3186a6f9dd5191924b961c56d4c01ca6b907aa\": container with ID starting with b6e307e08d883f07aed1537d0f3186a6f9dd5191924b961c56d4c01ca6b907aa not found: ID does not exist" containerID="b6e307e08d883f07aed1537d0f3186a6f9dd5191924b961c56d4c01ca6b907aa"
Jul 06 21:32:10.096989 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:10.096896    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e307e08d883f07aed1537d0f3186a6f9dd5191924b961c56d4c01ca6b907aa"} err="failed to get container status \"b6e307e08d883f07aed1537d0f3186a6f9dd5191924b961c56d4c01ca6b907aa\": rpc error: code = NotFound desc = could not find container \"b6e307e08d883f07aed1537d0f3186a6f9dd5191924b961c56d4c01ca6b907aa\": container with ID starting with b6e307e08d883f07aed1537d0f3186a6f9dd5191924b961c56d4c01ca6b907aa not found: ID does not exist"
Jul 06 21:32:11.879054 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:32:11.879020    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" path="/var/lib/kubelet/pods/74bad355-8e2d-4276-b23a-d713fcca540e/volumes"
Jul 06 21:34:22.572695 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:22.572615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" event={"ID":"8379cd1e-92b2-47d2-9e9d-cc6861e44030","Type":"ContainerStarted","Data":"c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67"}
Jul 06 21:34:22.572695 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:22.572675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" event={"ID":"8379cd1e-92b2-47d2-9e9d-cc6861e44030","Type":"ContainerStarted","Data":"77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146"}
Jul 06 21:34:22.573113 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:22.572790    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:34:22.620199 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:22.620148    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" podStartSLOduration=2.549796686 podStartE2EDuration="2m43.62013289s" totalImagesPullingTime="2m11.851213565s" totalInitContainerRuntime="29.219122639s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:34:22.619459761 +0000 UTC m=+1299.613558921" watchObservedRunningTime="2026-07-06 21:34:22.62013289 +0000 UTC m=+1299.614232050"
Jul 06 21:34:23.576299 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:23.576261    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:34:23.577757 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:23.577726    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 06 21:34:24.579602 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:24.579558    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 06 21:34:29.585709 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:29.585680    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:34:29.586291 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:29.586257    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 06 21:34:39.587445 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:39.587409    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:34:49.699630 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:49.699596    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"]
Jul 06 21:34:49.700079 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:49.699928    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="kserve-container" containerID="cri-o://77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146" gracePeriod=30
Jul 06 21:34:49.700079 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:49.699960    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="kube-rbac-proxy" containerID="cri-o://c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67" gracePeriod=30
Jul 06 21:34:50.573424 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.573391    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:34:50.578472 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.578452    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8379cd1e-92b2-47d2-9e9d-cc6861e44030-cabundle-cert\") pod \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") "
Jul 06 21:34:50.578559 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.578489    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kserve-provision-location\") pod \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") "
Jul 06 21:34:50.578559 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.578525    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kube-rbac-proxy-sar-config\") pod \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") "
Jul 06 21:34:50.578676 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.578646    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vq6k\" (UniqueName: \"kubernetes.io/projected/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kube-api-access-8vq6k\") pod \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") "
Jul 06 21:34:50.578731 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.578692    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8379cd1e-92b2-47d2-9e9d-cc6861e44030-proxy-tls\") pod \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\" (UID: \"8379cd1e-92b2-47d2-9e9d-cc6861e44030\") "
Jul 06 21:34:50.578879 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.578857    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8379cd1e-92b2-47d2-9e9d-cc6861e44030" (UID: "8379cd1e-92b2-47d2-9e9d-cc6861e44030"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:34:50.578879 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.578863    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8379cd1e-92b2-47d2-9e9d-cc6861e44030-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8379cd1e-92b2-47d2-9e9d-cc6861e44030" (UID: "8379cd1e-92b2-47d2-9e9d-cc6861e44030"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:34:50.580716 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.580687    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8379cd1e-92b2-47d2-9e9d-cc6861e44030-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8379cd1e-92b2-47d2-9e9d-cc6861e44030" (UID: "8379cd1e-92b2-47d2-9e9d-cc6861e44030"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:34:50.580805 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.580716    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kube-api-access-8vq6k" (OuterVolumeSpecName: "kube-api-access-8vq6k") pod "8379cd1e-92b2-47d2-9e9d-cc6861e44030" (UID: "8379cd1e-92b2-47d2-9e9d-cc6861e44030"). InnerVolumeSpecName "kube-api-access-8vq6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:34:50.583259 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.583213    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8379cd1e-92b2-47d2-9e9d-cc6861e44030" (UID: "8379cd1e-92b2-47d2-9e9d-cc6861e44030"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:34:50.673116 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.673086    2581 generic.go:356] "Generic (PLEG): container finished" podID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerID="c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67" exitCode=2
Jul 06 21:34:50.673116 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.673109    2581 generic.go:356] "Generic (PLEG): container finished" podID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerID="77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146" exitCode=0
Jul 06 21:34:50.673386 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.673128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" event={"ID":"8379cd1e-92b2-47d2-9e9d-cc6861e44030","Type":"ContainerDied","Data":"c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67"}
Jul 06 21:34:50.673386 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.673158    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"
Jul 06 21:34:50.673386 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.673177    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" event={"ID":"8379cd1e-92b2-47d2-9e9d-cc6861e44030","Type":"ContainerDied","Data":"77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146"}
Jul 06 21:34:50.673386 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.673198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w" event={"ID":"8379cd1e-92b2-47d2-9e9d-cc6861e44030","Type":"ContainerDied","Data":"cfc9bc6f585a66322d6667ae1a1538be54c26a5df55aba54693a233a90a17029"}
Jul 06 21:34:50.673386 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.673220    2581 scope.go:117] "RemoveContainer" containerID="c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67"
Jul 06 21:34:50.679284 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.679260    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vq6k\" (UniqueName: \"kubernetes.io/projected/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kube-api-access-8vq6k\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:34:50.679284 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.679283    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8379cd1e-92b2-47d2-9e9d-cc6861e44030-proxy-tls\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:34:50.679428 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.679293    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8379cd1e-92b2-47d2-9e9d-cc6861e44030-cabundle-cert\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:34:50.679428 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.679301    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kserve-provision-location\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:34:50.679428 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.679311    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8379cd1e-92b2-47d2-9e9d-cc6861e44030-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:34:50.681111 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.681092    2581 scope.go:117] "RemoveContainer" containerID="77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146"
Jul 06 21:34:50.688301 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.688285    2581 scope.go:117] "RemoveContainer" containerID="fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e"
Jul 06 21:34:50.695037 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.695015    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"]
Jul 06 21:34:50.695301 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.695284    2581 scope.go:117] "RemoveContainer" containerID="c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67"
Jul 06 21:34:50.695545 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:34:50.695528    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67\": container with ID starting with c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67 not found: ID does not exist" containerID="c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67"
Jul 06 21:34:50.695587 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.695555    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67"} err="failed to get container status \"c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67\": rpc error: code = NotFound desc = could not find container \"c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67\": container with ID starting with c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67 not found: ID does not exist"
Jul 06 21:34:50.695587 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.695572    2581 scope.go:117] "RemoveContainer" containerID="77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146"
Jul 06 21:34:50.695767 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:34:50.695753    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146\": container with ID starting with 77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146 not found: ID does not exist" containerID="77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146"
Jul 06 21:34:50.695802 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.695771    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146"} err="failed to get container status \"77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146\": rpc error: code = NotFound desc = could not find container \"77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146\": container with ID starting with 77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146 not found: ID does not exist"
Jul 06 21:34:50.695802 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.695784    2581 scope.go:117] "RemoveContainer" containerID="fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e"
Jul 06 21:34:50.695982 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:34:50.695967    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e\": container with ID starting with fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e not found: ID does not exist" containerID="fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e"
Jul 06 21:34:50.696026 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.695985    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e"} err="failed to get container status \"fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e\": rpc error: code = NotFound desc = could not find container \"fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e\": container with ID starting with fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e not found: ID does not exist"
Jul 06 21:34:50.696026 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.695998    2581 scope.go:117] "RemoveContainer" containerID="c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67"
Jul 06 21:34:50.696262 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.696223    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67"} err="failed to get container status \"c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67\": rpc error: code = NotFound desc = could not find container \"c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67\": container with ID starting with c5440018519d93b34bf1e0491c8c3344d8937f0bb8f5e17a2cdd9f7117e43c67 not found: ID does not exist"
Jul 06 21:34:50.696262 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.696260    2581 scope.go:117] "RemoveContainer" containerID="77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146"
Jul 06 21:34:50.696513 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.696495    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146"} err="failed to get container status \"77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146\": rpc error: code = NotFound desc = could not find container \"77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146\": container with ID starting with 77f493b9ed9e0478cc5d66dc7f10fb24533aea16319e09941d3d05cf086d1146 not found: ID does not exist"
Jul 06 21:34:50.696561 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.696514    2581 scope.go:117] "RemoveContainer" containerID="fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e"
Jul 06 21:34:50.696715 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.696698    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e"} err="failed to get container status \"fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e\": rpc error: code = NotFound desc = could not find container \"fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e\": container with ID starting with fac272383a0f11b6d69ad456079a31443d646c5a2b21cf6afdf64d9925f0556e not found: ID does not exist"
Jul 06 21:34:50.700124 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:50.700104    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lrx4w"]
Jul 06 21:34:51.877931 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:51.877902    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" path="/var/lib/kubelet/pods/8379cd1e-92b2-47d2-9e9d-cc6861e44030/volumes"
Jul 06 21:34:55.048839 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.048803    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"]
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049079    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049090    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049099    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3221ede-d447-45e5-acba-8bab4b29b92f" containerName="kserve-container"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049104    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3221ede-d447-45e5-acba-8bab4b29b92f" containerName="kserve-container"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049114    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="storage-initializer"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049120    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="storage-initializer"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049125    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="kube-rbac-proxy"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049130    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="kube-rbac-proxy"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049139    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="storage-initializer"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049144    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="storage-initializer"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049149    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kube-rbac-proxy"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049155    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kube-rbac-proxy"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049161    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="kserve-container"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049166    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="kserve-container"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049175    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3221ede-d447-45e5-acba-8bab4b29b92f" containerName="kube-rbac-proxy"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049181    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3221ede-d447-45e5-acba-8bab4b29b92f" containerName="kube-rbac-proxy"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049188    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049193    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049246    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kube-rbac-proxy"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049253    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="kserve-container"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049259    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="agent"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049267    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3221ede-d447-45e5-acba-8bab4b29b92f" containerName="kserve-container"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049272    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="74bad355-8e2d-4276-b23a-d713fcca540e" containerName="kserve-container"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049277    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8379cd1e-92b2-47d2-9e9d-cc6861e44030" containerName="kube-rbac-proxy"
Jul 06 21:34:55.049280 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.049283    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3221ede-d447-45e5-acba-8bab4b29b92f" containerName="kube-rbac-proxy"
Jul 06 21:34:55.085651 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.085618    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"]
Jul 06 21:34:55.085779 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.085750    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.088596 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.088574    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:34:55.088732 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.088577    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 06 21:34:55.088732 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.088577    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pnnb5\""
Jul 06 21:34:55.089304 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.089290    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:34:55.089364 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.089302    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 06 21:34:55.089513 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.089500    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:34:55.089601 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.089579    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 21:34:55.109362 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.109332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b63a58ca-fa61-4643-b896-b722fb7c8c06-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.109485 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.109400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b63a58ca-fa61-4643-b896-b722fb7c8c06-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.109485 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.109442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b63a58ca-fa61-4643-b896-b722fb7c8c06-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.109485 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.109473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wn84\" (UniqueName: \"kubernetes.io/projected/b63a58ca-fa61-4643-b896-b722fb7c8c06-kube-api-access-7wn84\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.109633 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.109514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b63a58ca-fa61-4643-b896-b722fb7c8c06-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.210803 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.210764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b63a58ca-fa61-4643-b896-b722fb7c8c06-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.210803 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.210803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b63a58ca-fa61-4643-b896-b722fb7c8c06-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.211067 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.210821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wn84\" (UniqueName: \"kubernetes.io/projected/b63a58ca-fa61-4643-b896-b722fb7c8c06-kube-api-access-7wn84\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.211067 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.210844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b63a58ca-fa61-4643-b896-b722fb7c8c06-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.211067 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.210884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b63a58ca-fa61-4643-b896-b722fb7c8c06-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.211067 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:34:55.210935    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 06 21:34:55.211067 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:34:55.211015    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b63a58ca-fa61-4643-b896-b722fb7c8c06-proxy-tls podName:b63a58ca-fa61-4643-b896-b722fb7c8c06 nodeName:}" failed. No retries permitted until 2026-07-06 21:34:55.710993885 +0000 UTC m=+1332.705093037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b63a58ca-fa61-4643-b896-b722fb7c8c06-proxy-tls") pod "isvc-autogluon-v2-predictor-67b849d456-zmv5r" (UID: "b63a58ca-fa61-4643-b896-b722fb7c8c06") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 06 21:34:55.211366 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.211348    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b63a58ca-fa61-4643-b896-b722fb7c8c06-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.211667 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.211645    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b63a58ca-fa61-4643-b896-b722fb7c8c06-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.211701 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.211645    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b63a58ca-fa61-4643-b896-b722fb7c8c06-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.219603 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.219585    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wn84\" (UniqueName: \"kubernetes.io/projected/b63a58ca-fa61-4643-b896-b722fb7c8c06-kube-api-access-7wn84\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.713765 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.713729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b63a58ca-fa61-4643-b896-b722fb7c8c06-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.716122 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.716102    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b63a58ca-fa61-4643-b896-b722fb7c8c06-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-zmv5r\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:55.996705 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:55.996620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:34:56.161166 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:56.161145    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"]
Jul 06 21:34:56.163719 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:34:56.163688    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63a58ca_fa61_4643_b896_b722fb7c8c06.slice/crio-64fc8ed20e37803f275bbf5dba63ebd22a005520c97b4baf45d29ddd8ea4b3c0 WatchSource:0}: Error finding container 64fc8ed20e37803f275bbf5dba63ebd22a005520c97b4baf45d29ddd8ea4b3c0: Status 404 returned error can't find the container with id 64fc8ed20e37803f275bbf5dba63ebd22a005520c97b4baf45d29ddd8ea4b3c0
Jul 06 21:34:56.698766 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:56.698728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r" event={"ID":"b63a58ca-fa61-4643-b896-b722fb7c8c06","Type":"ContainerStarted","Data":"c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e"}
Jul 06 21:34:56.698926 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:34:56.698801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r" event={"ID":"b63a58ca-fa61-4643-b896-b722fb7c8c06","Type":"ContainerStarted","Data":"64fc8ed20e37803f275bbf5dba63ebd22a005520c97b4baf45d29ddd8ea4b3c0"}
Jul 06 21:35:18.775437 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:35:18.775401    2581 generic.go:356] "Generic (PLEG): container finished" podID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerID="c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e" exitCode=0
Jul 06 21:35:18.775837 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:35:18.775476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r" event={"ID":"b63a58ca-fa61-4643-b896-b722fb7c8c06","Type":"ContainerDied","Data":"c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e"}
Jul 06 21:35:19.781241 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:35:19.781195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r" event={"ID":"b63a58ca-fa61-4643-b896-b722fb7c8c06","Type":"ContainerStarted","Data":"9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db"}
Jul 06 21:35:19.781673 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:35:19.781269    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r" event={"ID":"b63a58ca-fa61-4643-b896-b722fb7c8c06","Type":"ContainerStarted","Data":"520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539"}
Jul 06 21:35:19.781673 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:35:19.781453    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:35:19.834001 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:35:19.833948    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r" podStartSLOduration=3.102585686 podStartE2EDuration="24.833930653s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.731344967s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:35:19.832952397 +0000 UTC m=+1356.827051571" watchObservedRunningTime="2026-07-06 21:35:19.833930653 +0000 UTC m=+1356.828029811"
Jul 06 21:35:20.785153 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:35:20.785122    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:35:26.794753 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:35:26.794724    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:36:56.799679 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:36:56.799638    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:37:05.161118 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:05.161087    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"]
Jul 06 21:37:05.161613 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:05.161365    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r" podUID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerName="kserve-container" containerID="cri-o://520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539" gracePeriod=30
Jul 06 21:37:05.161613 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:05.161572    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r" podUID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerName="kube-rbac-proxy" containerID="cri-o://9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db" gracePeriod=30
Jul 06 21:37:06.020215 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.020191    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:37:06.081748 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.081711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b63a58ca-fa61-4643-b896-b722fb7c8c06-proxy-tls\") pod \"b63a58ca-fa61-4643-b896-b722fb7c8c06\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") "
Jul 06 21:37:06.081934 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.081764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b63a58ca-fa61-4643-b896-b722fb7c8c06-cabundle-cert\") pod \"b63a58ca-fa61-4643-b896-b722fb7c8c06\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") "
Jul 06 21:37:06.081934 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.081795    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wn84\" (UniqueName: \"kubernetes.io/projected/b63a58ca-fa61-4643-b896-b722fb7c8c06-kube-api-access-7wn84\") pod \"b63a58ca-fa61-4643-b896-b722fb7c8c06\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") "
Jul 06 21:37:06.081934 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.081850    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b63a58ca-fa61-4643-b896-b722fb7c8c06-kserve-provision-location\") pod \"b63a58ca-fa61-4643-b896-b722fb7c8c06\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") "
Jul 06 21:37:06.081934 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.081866    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b63a58ca-fa61-4643-b896-b722fb7c8c06-kube-rbac-proxy-sar-config\") pod \"b63a58ca-fa61-4643-b896-b722fb7c8c06\" (UID: \"b63a58ca-fa61-4643-b896-b722fb7c8c06\") "
Jul 06 21:37:06.082269 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.082213    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b63a58ca-fa61-4643-b896-b722fb7c8c06-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b63a58ca-fa61-4643-b896-b722fb7c8c06" (UID: "b63a58ca-fa61-4643-b896-b722fb7c8c06"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:37:06.082379 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.082263    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b63a58ca-fa61-4643-b896-b722fb7c8c06-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b63a58ca-fa61-4643-b896-b722fb7c8c06" (UID: "b63a58ca-fa61-4643-b896-b722fb7c8c06"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:37:06.083940 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.083914    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b63a58ca-fa61-4643-b896-b722fb7c8c06-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b63a58ca-fa61-4643-b896-b722fb7c8c06" (UID: "b63a58ca-fa61-4643-b896-b722fb7c8c06"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:37:06.084261 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.084207    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63a58ca-fa61-4643-b896-b722fb7c8c06-kube-api-access-7wn84" (OuterVolumeSpecName: "kube-api-access-7wn84") pod "b63a58ca-fa61-4643-b896-b722fb7c8c06" (UID: "b63a58ca-fa61-4643-b896-b722fb7c8c06"). InnerVolumeSpecName "kube-api-access-7wn84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:37:06.087246 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.087201    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63a58ca-fa61-4643-b896-b722fb7c8c06-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b63a58ca-fa61-4643-b896-b722fb7c8c06" (UID: "b63a58ca-fa61-4643-b896-b722fb7c8c06"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:37:06.162154 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.162121    2581 generic.go:356] "Generic (PLEG): container finished" podID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerID="9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db" exitCode=2
Jul 06 21:37:06.162154 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.162145    2581 generic.go:356] "Generic (PLEG): container finished" podID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerID="520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539" exitCode=0
Jul 06 21:37:06.162661 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.162179    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"
Jul 06 21:37:06.162661 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.162192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r" event={"ID":"b63a58ca-fa61-4643-b896-b722fb7c8c06","Type":"ContainerDied","Data":"9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db"}
Jul 06 21:37:06.162661 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.162255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r" event={"ID":"b63a58ca-fa61-4643-b896-b722fb7c8c06","Type":"ContainerDied","Data":"520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539"}
Jul 06 21:37:06.162661 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.162277    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r" event={"ID":"b63a58ca-fa61-4643-b896-b722fb7c8c06","Type":"ContainerDied","Data":"64fc8ed20e37803f275bbf5dba63ebd22a005520c97b4baf45d29ddd8ea4b3c0"}
Jul 06 21:37:06.162661 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.162301    2581 scope.go:117] "RemoveContainer" containerID="9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db"
Jul 06 21:37:06.171313 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.171294    2581 scope.go:117] "RemoveContainer" containerID="520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539"
Jul 06 21:37:06.178195 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.178177    2581 scope.go:117] "RemoveContainer" containerID="c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e"
Jul 06 21:37:06.182274 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.182255    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b63a58ca-fa61-4643-b896-b722fb7c8c06-kserve-provision-location\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:37:06.182274 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.182275    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b63a58ca-fa61-4643-b896-b722fb7c8c06-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:37:06.182387 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.182285    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b63a58ca-fa61-4643-b896-b722fb7c8c06-proxy-tls\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:37:06.182387 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.182294    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b63a58ca-fa61-4643-b896-b722fb7c8c06-cabundle-cert\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:37:06.182387 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.182303    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wn84\" (UniqueName: \"kubernetes.io/projected/b63a58ca-fa61-4643-b896-b722fb7c8c06-kube-api-access-7wn84\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:37:06.185072 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.185046    2581 scope.go:117] "RemoveContainer" containerID="9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db"
Jul 06 21:37:06.185386 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:37:06.185364    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db\": container with ID starting with 9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db not found: ID does not exist" containerID="9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db"
Jul 06 21:37:06.185461 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.185394    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db"} err="failed to get container status \"9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db\": rpc error: code = NotFound desc = could not find container \"9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db\": container with ID starting with 9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db not found: ID does not exist"
Jul 06 21:37:06.185461 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.185440    2581 scope.go:117] "RemoveContainer" containerID="520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539"
Jul 06 21:37:06.185708 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:37:06.185688    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539\": container with ID starting with 520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539 not found: ID does not exist" containerID="520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539"
Jul 06 21:37:06.185756 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.185715    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539"} err="failed to get container status \"520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539\": rpc error: code = NotFound desc = could not find container \"520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539\": container with ID starting with 520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539 not found: ID does not exist"
Jul 06 21:37:06.185756 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.185732    2581 scope.go:117] "RemoveContainer" containerID="c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e"
Jul 06 21:37:06.185994 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:37:06.185975    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e\": container with ID starting with c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e not found: ID does not exist" containerID="c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e"
Jul 06 21:37:06.186067 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.186003    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e"} err="failed to get container status \"c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e\": rpc error: code = NotFound desc = could not find container \"c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e\": container with ID starting with c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e not found: ID does not exist"
Jul 06 21:37:06.186067 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.186024    2581 scope.go:117] "RemoveContainer" containerID="9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db"
Jul 06 21:37:06.186067 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.186040    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"]
Jul 06 21:37:06.186277 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.186257    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db"} err="failed to get container status \"9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db\": rpc error: code = NotFound desc = could not find container \"9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db\": container with ID starting with 9c4cfd121c98e9c6dbea02db5e0d435ff6854fea41e6bdf3f4632f6dd82da1db not found: ID does not exist"
Jul 06 21:37:06.186323 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.186279    2581 scope.go:117] "RemoveContainer" containerID="520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539"
Jul 06 21:37:06.186486 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.186470    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539"} err="failed to get container status \"520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539\": rpc error: code = NotFound desc = could not find container \"520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539\": container with ID starting with 520bf8ac25f7557ab219950269110421ce842440bf4719ece2628f25cce28539 not found: ID does not exist"
Jul 06 21:37:06.186527 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.186487    2581 scope.go:117] "RemoveContainer" containerID="c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e"
Jul 06 21:37:06.186679 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.186662    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e"} err="failed to get container status \"c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e\": rpc error: code = NotFound desc = could not find container \"c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e\": container with ID starting with c6a585c70563fdd4780ff58983f9f45e9c33537a9a75972582b7665d180a7b1e not found: ID does not exist"
Jul 06 21:37:06.189845 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:06.189824    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-zmv5r"]
Jul 06 21:37:07.877120 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:07.877084    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b63a58ca-fa61-4643-b896-b722fb7c8c06" path="/var/lib/kubelet/pods/b63a58ca-fa61-4643-b896-b722fb7c8c06/volumes"
Jul 06 21:37:10.299483 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.299447    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"]
Jul 06 21:37:10.299853 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.299715    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerName="storage-initializer"
Jul 06 21:37:10.299853 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.299729    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerName="storage-initializer"
Jul 06 21:37:10.299853 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.299741    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerName="kserve-container"
Jul 06 21:37:10.299853 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.299751    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerName="kserve-container"
Jul 06 21:37:10.299853 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.299763    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerName="kube-rbac-proxy"
Jul 06 21:37:10.299853 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.299772    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerName="kube-rbac-proxy"
Jul 06 21:37:10.300039 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.299857    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerName="kube-rbac-proxy"
Jul 06 21:37:10.300039 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.299866    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b63a58ca-fa61-4643-b896-b722fb7c8c06" containerName="kserve-container"
Jul 06 21:37:10.303113 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.303097    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.305398 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.305376    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:37:10.305533 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.305376    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pnnb5\""
Jul 06 21:37:10.305533 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.305377    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 06 21:37:10.305533 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.305455    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:37:10.305694 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.305549    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:37:10.305694 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.305654    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 06 21:37:10.305810 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.305794    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 21:37:10.315509 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.315487    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"]
Jul 06 21:37:10.414894 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.414858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.415070 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.414900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.415070 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.415001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4wx2\" (UniqueName: \"kubernetes.io/projected/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kube-api-access-x4wx2\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.415070 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.415051    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.415173 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.415075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.516261 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.516208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.516424 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.516272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.516424 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.516302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.516424 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.516343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4wx2\" (UniqueName: \"kubernetes.io/projected/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kube-api-access-x4wx2\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.516424 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.516373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.516815 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.516791    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.516964 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.516937    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.517062 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.516974    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.518787 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.518769    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.523889 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.523859    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4wx2\" (UniqueName: \"kubernetes.io/projected/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kube-api-access-x4wx2\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.613615 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.613523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:10.741143 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:10.741121    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"]
Jul 06 21:37:10.743771 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:37:10.743746    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be03b3a_1b2f_4173_bf09_e0b2b4f1a8e7.slice/crio-bd1d810f741da5f92db4c7f35d231d2d5997e4107451d47247e2d2e3a218b1d4 WatchSource:0}: Error finding container bd1d810f741da5f92db4c7f35d231d2d5997e4107451d47247e2d2e3a218b1d4: Status 404 returned error can't find the container with id bd1d810f741da5f92db4c7f35d231d2d5997e4107451d47247e2d2e3a218b1d4
Jul 06 21:37:11.181925 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:11.181885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm" event={"ID":"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7","Type":"ContainerStarted","Data":"84a2900283a29b751d14eadd37d04bf9f61946de2795989894bb72a0aa31d0f2"}
Jul 06 21:37:11.182090 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:11.181963    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm" event={"ID":"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7","Type":"ContainerStarted","Data":"bd1d810f741da5f92db4c7f35d231d2d5997e4107451d47247e2d2e3a218b1d4"}
Jul 06 21:37:16.201360 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:16.201328    2581 generic.go:356] "Generic (PLEG): container finished" podID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerID="84a2900283a29b751d14eadd37d04bf9f61946de2795989894bb72a0aa31d0f2" exitCode=0
Jul 06 21:37:16.201743 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:16.201385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm" event={"ID":"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7","Type":"ContainerDied","Data":"84a2900283a29b751d14eadd37d04bf9f61946de2795989894bb72a0aa31d0f2"}
Jul 06 21:37:17.207786 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:17.207754    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm" event={"ID":"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7","Type":"ContainerStarted","Data":"270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585"}
Jul 06 21:37:17.208159 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:17.207814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm" event={"ID":"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7","Type":"ContainerStarted","Data":"1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108"}
Jul 06 21:37:17.208159 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:17.208007    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:17.229411 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:17.229360    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm" podStartSLOduration=2.753118889 podStartE2EDuration="7.229342744s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.476223855s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:37:17.227752146 +0000 UTC m=+1474.221851308" watchObservedRunningTime="2026-07-06 21:37:17.229342744 +0000 UTC m=+1474.223441904"
Jul 06 21:37:18.211712 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:18.211681    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:37:24.220632 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:37:24.220604    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:38:55.878015 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:38:55.877985    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:39:00.432900 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:00.432857    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"]
Jul 06 21:39:00.433393 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:00.433157    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm" podUID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerName="kserve-container" containerID="cri-o://1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108" gracePeriod=30
Jul 06 21:39:00.433393 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:00.433184    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm" podUID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerName="kube-rbac-proxy" containerID="cri-o://270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585" gracePeriod=30
Jul 06 21:39:00.571991 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:00.571956    2581 generic.go:356] "Generic (PLEG): container finished" podID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerID="270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585" exitCode=2
Jul 06 21:39:00.572151 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:00.572024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm" event={"ID":"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7","Type":"ContainerDied","Data":"270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585"}
Jul 06 21:39:01.292297 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.292272    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:39:01.440185 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.440153    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4wx2\" (UniqueName: \"kubernetes.io/projected/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kube-api-access-x4wx2\") pod \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") "
Jul 06 21:39:01.440698 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.440194    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-cabundle-cert\") pod \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") "
Jul 06 21:39:01.440698 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.440222    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-proxy-tls\") pod \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") "
Jul 06 21:39:01.440698 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.440276    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kserve-provision-location\") pod \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") "
Jul 06 21:39:01.440698 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.440306    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kube-rbac-proxy-sar-config\") pod \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\" (UID: \"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7\") "
Jul 06 21:39:01.440698 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.440638    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" (UID: "3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:39:01.440698 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.440671    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" (UID: "3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:39:01.442408 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.442384    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" (UID: "3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:39:01.442510 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.442387    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kube-api-access-x4wx2" (OuterVolumeSpecName: "kube-api-access-x4wx2") pod "3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" (UID: "3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7"). InnerVolumeSpecName "kube-api-access-x4wx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:39:01.444677 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.444645    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" (UID: "3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:39:01.540907 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.540868    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4wx2\" (UniqueName: \"kubernetes.io/projected/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kube-api-access-x4wx2\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:39:01.540907 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.540901    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-cabundle-cert\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:39:01.540907 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.540911    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-proxy-tls\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:39:01.540907 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.540919    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kserve-provision-location\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:39:01.541167 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.540928    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:39:01.578268 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.578222    2581 generic.go:356] "Generic (PLEG): container finished" podID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerID="1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108" exitCode=0
Jul 06 21:39:01.578398 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.578271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm" event={"ID":"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7","Type":"ContainerDied","Data":"1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108"}
Jul 06 21:39:01.578398 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.578298    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"
Jul 06 21:39:01.578398 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.578333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm" event={"ID":"3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7","Type":"ContainerDied","Data":"bd1d810f741da5f92db4c7f35d231d2d5997e4107451d47247e2d2e3a218b1d4"}
Jul 06 21:39:01.578398 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.578362    2581 scope.go:117] "RemoveContainer" containerID="270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585"
Jul 06 21:39:01.586393 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.586376    2581 scope.go:117] "RemoveContainer" containerID="1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108"
Jul 06 21:39:01.593812 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.593787    2581 scope.go:117] "RemoveContainer" containerID="84a2900283a29b751d14eadd37d04bf9f61946de2795989894bb72a0aa31d0f2"
Jul 06 21:39:01.599744 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.599723    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"]
Jul 06 21:39:01.601564 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.601547    2581 scope.go:117] "RemoveContainer" containerID="270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585"
Jul 06 21:39:01.601877 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:39:01.601856    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585\": container with ID starting with 270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585 not found: ID does not exist" containerID="270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585"
Jul 06 21:39:01.601932 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.601886    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585"} err="failed to get container status \"270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585\": rpc error: code = NotFound desc = could not find container \"270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585\": container with ID starting with 270ec47122a77f46e313ca5ebabd9c92429bbde0546bce333c9fbf9252014585 not found: ID does not exist"
Jul 06 21:39:01.601932 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.601906    2581 scope.go:117] "RemoveContainer" containerID="1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108"
Jul 06 21:39:01.602156 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:39:01.602138    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108\": container with ID starting with 1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108 not found: ID does not exist" containerID="1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108"
Jul 06 21:39:01.602206 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.602163    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108"} err="failed to get container status \"1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108\": rpc error: code = NotFound desc = could not find container \"1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108\": container with ID starting with 1ed4c970d52c0a7b1d69e9a36adc99fe99f1fbda320c2bf28aa6bea9a198a108 not found: ID does not exist"
Jul 06 21:39:01.602206 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.602183    2581 scope.go:117] "RemoveContainer" containerID="84a2900283a29b751d14eadd37d04bf9f61946de2795989894bb72a0aa31d0f2"
Jul 06 21:39:01.602453 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:39:01.602439    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a2900283a29b751d14eadd37d04bf9f61946de2795989894bb72a0aa31d0f2\": container with ID starting with 84a2900283a29b751d14eadd37d04bf9f61946de2795989894bb72a0aa31d0f2 not found: ID does not exist" containerID="84a2900283a29b751d14eadd37d04bf9f61946de2795989894bb72a0aa31d0f2"
Jul 06 21:39:01.602501 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.602455    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a2900283a29b751d14eadd37d04bf9f61946de2795989894bb72a0aa31d0f2"} err="failed to get container status \"84a2900283a29b751d14eadd37d04bf9f61946de2795989894bb72a0aa31d0f2\": rpc error: code = NotFound desc = could not find container \"84a2900283a29b751d14eadd37d04bf9f61946de2795989894bb72a0aa31d0f2\": container with ID starting with 84a2900283a29b751d14eadd37d04bf9f61946de2795989894bb72a0aa31d0f2 not found: ID does not exist"
Jul 06 21:39:01.605652 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.605634    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-s7cnm"]
Jul 06 21:39:01.878598 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:01.878521    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" path="/var/lib/kubelet/pods/3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7/volumes"
Jul 06 21:39:05.581591 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.581553    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"]
Jul 06 21:39:05.581969 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.581805    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerName="kserve-container"
Jul 06 21:39:05.581969 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.581815    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerName="kserve-container"
Jul 06 21:39:05.581969 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.581824    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerName="storage-initializer"
Jul 06 21:39:05.581969 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.581829    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerName="storage-initializer"
Jul 06 21:39:05.581969 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.581841    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerName="kube-rbac-proxy"
Jul 06 21:39:05.581969 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.581846    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerName="kube-rbac-proxy"
Jul 06 21:39:05.581969 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.581904    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerName="kube-rbac-proxy"
Jul 06 21:39:05.581969 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.581914    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3be03b3a-1b2f-4173-bf09-e0b2b4f1a8e7" containerName="kserve-container"
Jul 06 21:39:05.586297 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.586275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.588715 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.588692    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pnnb5\""
Jul 06 21:39:05.588842 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.588783    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 06 21:39:05.588967 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.588697    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:39:05.588967 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.588744    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 06 21:39:05.588967 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.588745    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 21:39:05.588967 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.588921    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:39:05.588967 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.588864    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:39:05.596178 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.596154    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"]
Jul 06 21:39:05.774980 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.774939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfqb\" (UniqueName: \"kubernetes.io/projected/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kube-api-access-jrfqb\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.775157 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.775004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.775157 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.775048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.775157 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.775072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.775297 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.775156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.875627 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.875557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.875627 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.875591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfqb\" (UniqueName: \"kubernetes.io/projected/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kube-api-access-jrfqb\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.875783 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.875642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.875783 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.875674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.875783 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.875697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.876011 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.875987    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.876320 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.876293    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.876432 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.876404    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.878145 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.878122    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.885068 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.885050    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfqb\" (UniqueName: \"kubernetes.io/projected/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kube-api-access-jrfqb\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:05.897206 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:05.897178    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:06.027851 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:06.027825    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"]
Jul 06 21:39:06.030176 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:39:06.030148    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6063bdc9_8fc4_4be0_a2b3_4667e26ed9c2.slice/crio-82c37007364209f4ebbad2135ecd713e059068ffd6e09c9031ebf3e55d762cfa WatchSource:0}: Error finding container 82c37007364209f4ebbad2135ecd713e059068ffd6e09c9031ebf3e55d762cfa: Status 404 returned error can't find the container with id 82c37007364209f4ebbad2135ecd713e059068ffd6e09c9031ebf3e55d762cfa
Jul 06 21:39:06.596534 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:06.596498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r" event={"ID":"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2","Type":"ContainerStarted","Data":"9acb1b3cec5a6dc0b824c67c5f33c1876f4a1e2295fb345ae02aa8a900cbc52e"}
Jul 06 21:39:06.596897 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:06.596556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r" event={"ID":"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2","Type":"ContainerStarted","Data":"82c37007364209f4ebbad2135ecd713e059068ffd6e09c9031ebf3e55d762cfa"}
Jul 06 21:39:33.692982 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:33.692950    2581 generic.go:356] "Generic (PLEG): container finished" podID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerID="9acb1b3cec5a6dc0b824c67c5f33c1876f4a1e2295fb345ae02aa8a900cbc52e" exitCode=0
Jul 06 21:39:33.693405 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:33.693023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r" event={"ID":"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2","Type":"ContainerDied","Data":"9acb1b3cec5a6dc0b824c67c5f33c1876f4a1e2295fb345ae02aa8a900cbc52e"}
Jul 06 21:39:34.698895 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:34.698859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r" event={"ID":"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2","Type":"ContainerStarted","Data":"812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298"}
Jul 06 21:39:34.699404 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:34.698916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r" event={"ID":"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2","Type":"ContainerStarted","Data":"a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5"}
Jul 06 21:39:34.699404 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:34.699110    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:34.699404 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:34.699162    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:39:34.717919 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:34.717866    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r" podStartSLOduration=2.727665911 podStartE2EDuration="29.717848532s" totalImagesPullingTime="0s" totalInitContainerRuntime="26.990182621s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:39:34.716922618 +0000 UTC m=+1611.711021798" watchObservedRunningTime="2026-07-06 21:39:34.717848532 +0000 UTC m=+1611.711947691"
Jul 06 21:39:40.708336 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:39:40.708304    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:41:04.877735 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:04.877701    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:41:05.722924 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:05.722889    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"]
Jul 06 21:41:05.723218 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:05.723179    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r" podUID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerName="kserve-container" containerID="cri-o://a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5" gracePeriod=30
Jul 06 21:41:05.723318 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:05.723217    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r" podUID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerName="kube-rbac-proxy" containerID="cri-o://812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298" gracePeriod=30
Jul 06 21:41:06.020698 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.020613    2581 generic.go:356] "Generic (PLEG): container finished" podID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerID="812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298" exitCode=2
Jul 06 21:41:06.021035 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.020686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r" event={"ID":"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2","Type":"ContainerDied","Data":"812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298"}
Jul 06 21:41:06.565249 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.565212    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:41:06.656451 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.656353    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrfqb\" (UniqueName: \"kubernetes.io/projected/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kube-api-access-jrfqb\") pod \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") "
Jul 06 21:41:06.656451 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.656401    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kube-rbac-proxy-sar-config\") pod \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") "
Jul 06 21:41:06.656451 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.656437    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-proxy-tls\") pod \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") "
Jul 06 21:41:06.656685 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.656457    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-cabundle-cert\") pod \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") "
Jul 06 21:41:06.656685 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.656479    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kserve-provision-location\") pod \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\" (UID: \"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2\") "
Jul 06 21:41:06.656838 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.656809    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" (UID: "6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:41:06.656912 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.656885    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" (UID: "6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:41:06.658589 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.658568    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kube-api-access-jrfqb" (OuterVolumeSpecName: "kube-api-access-jrfqb") pod "6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" (UID: "6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2"). InnerVolumeSpecName "kube-api-access-jrfqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:41:06.658654 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.658567    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" (UID: "6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:41:06.661482 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.661462    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" (UID: "6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:06.757930 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.757890    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:41:06.757930 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.757924    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-proxy-tls\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:41:06.757930 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.757936    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-cabundle-cert\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:41:06.758208 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.757947    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kserve-provision-location\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:41:06.758208 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:06.757961    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrfqb\" (UniqueName: \"kubernetes.io/projected/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2-kube-api-access-jrfqb\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:41:07.026271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.026169    2581 generic.go:356] "Generic (PLEG): container finished" podID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerID="a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5" exitCode=0
Jul 06 21:41:07.026271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.026245    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"
Jul 06 21:41:07.026271 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.026254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r" event={"ID":"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2","Type":"ContainerDied","Data":"a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5"}
Jul 06 21:41:07.026764 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.026305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r" event={"ID":"6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2","Type":"ContainerDied","Data":"82c37007364209f4ebbad2135ecd713e059068ffd6e09c9031ebf3e55d762cfa"}
Jul 06 21:41:07.026764 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.026331    2581 scope.go:117] "RemoveContainer" containerID="812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298"
Jul 06 21:41:07.034666 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.034645    2581 scope.go:117] "RemoveContainer" containerID="a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5"
Jul 06 21:41:07.041682 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.041663    2581 scope.go:117] "RemoveContainer" containerID="9acb1b3cec5a6dc0b824c67c5f33c1876f4a1e2295fb345ae02aa8a900cbc52e"
Jul 06 21:41:07.047945 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.047915    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"]
Jul 06 21:41:07.049430 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.049409    2581 scope.go:117] "RemoveContainer" containerID="812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298"
Jul 06 21:41:07.049590 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.049570    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-tqc9r"]
Jul 06 21:41:07.049683 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:41:07.049668    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298\": container with ID starting with 812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298 not found: ID does not exist" containerID="812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298"
Jul 06 21:41:07.049722 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.049691    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298"} err="failed to get container status \"812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298\": rpc error: code = NotFound desc = could not find container \"812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298\": container with ID starting with 812ceeb42517bd2c9b18f7ec90851d108b8e9d71af29ef323d050a0cf8097298 not found: ID does not exist"
Jul 06 21:41:07.049722 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.049707    2581 scope.go:117] "RemoveContainer" containerID="a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5"
Jul 06 21:41:07.049945 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:41:07.049928    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5\": container with ID starting with a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5 not found: ID does not exist" containerID="a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5"
Jul 06 21:41:07.050003 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.049953    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5"} err="failed to get container status \"a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5\": rpc error: code = NotFound desc = could not find container \"a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5\": container with ID starting with a71f3d860d5817ba5e3acb990d6aca9962e9cd65a2a900cbe718a664bb7dadc5 not found: ID does not exist"
Jul 06 21:41:07.050003 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.049969    2581 scope.go:117] "RemoveContainer" containerID="9acb1b3cec5a6dc0b824c67c5f33c1876f4a1e2295fb345ae02aa8a900cbc52e"
Jul 06 21:41:07.050176 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:41:07.050159    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9acb1b3cec5a6dc0b824c67c5f33c1876f4a1e2295fb345ae02aa8a900cbc52e\": container with ID starting with 9acb1b3cec5a6dc0b824c67c5f33c1876f4a1e2295fb345ae02aa8a900cbc52e not found: ID does not exist" containerID="9acb1b3cec5a6dc0b824c67c5f33c1876f4a1e2295fb345ae02aa8a900cbc52e"
Jul 06 21:41:07.050214 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.050182    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9acb1b3cec5a6dc0b824c67c5f33c1876f4a1e2295fb345ae02aa8a900cbc52e"} err="failed to get container status \"9acb1b3cec5a6dc0b824c67c5f33c1876f4a1e2295fb345ae02aa8a900cbc52e\": rpc error: code = NotFound desc = could not find container \"9acb1b3cec5a6dc0b824c67c5f33c1876f4a1e2295fb345ae02aa8a900cbc52e\": container with ID starting with 9acb1b3cec5a6dc0b824c67c5f33c1876f4a1e2295fb345ae02aa8a900cbc52e not found: ID does not exist"
Jul 06 21:41:07.877211 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:07.877168    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" path="/var/lib/kubelet/pods/6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2/volumes"
Jul 06 21:41:10.885600 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.885564    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"]
Jul 06 21:41:10.885966 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.885817    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerName="kube-rbac-proxy"
Jul 06 21:41:10.885966 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.885827    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerName="kube-rbac-proxy"
Jul 06 21:41:10.885966 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.885837    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerName="storage-initializer"
Jul 06 21:41:10.885966 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.885843    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerName="storage-initializer"
Jul 06 21:41:10.885966 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.885854    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerName="kserve-container"
Jul 06 21:41:10.885966 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.885860    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerName="kserve-container"
Jul 06 21:41:10.885966 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.885913    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerName="kube-rbac-proxy"
Jul 06 21:41:10.885966 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.885920    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6063bdc9-8fc4-4be0-a2b3-4667e26ed9c2" containerName="kserve-container"
Jul 06 21:41:10.890722 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.890702    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:10.893202 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.893177    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 21:41:10.893364 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.893177    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 06 21:41:10.893364 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.893346    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pnnb5\""
Jul 06 21:41:10.893480 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.893362    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:41:10.893480 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.893398    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 06 21:41:10.893480 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.893444    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:41:10.893979 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.893964    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:41:10.901425 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.901399    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"]
Jul 06 21:41:10.988059 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.988023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:10.988285 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.988066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:10.988285 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.988105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a041dab-85bd-4b08-86dc-9d078ba1cf60-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:10.988285 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.988140    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpnxp\" (UniqueName: \"kubernetes.io/projected/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kube-api-access-cpnxp\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:10.988285 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:10.988207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a041dab-85bd-4b08-86dc-9d078ba1cf60-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.089410 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.089372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a041dab-85bd-4b08-86dc-9d078ba1cf60-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.089410 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.089416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpnxp\" (UniqueName: \"kubernetes.io/projected/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kube-api-access-cpnxp\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.089666 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.089448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a041dab-85bd-4b08-86dc-9d078ba1cf60-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.089666 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.089475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.089666 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.089494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.089666 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:41:11.089612    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 06 21:41:11.089882 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:41:11.089692    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a041dab-85bd-4b08-86dc-9d078ba1cf60-proxy-tls podName:4a041dab-85bd-4b08-86dc-9d078ba1cf60 nodeName:}" failed. No retries permitted until 2026-07-06 21:41:11.589671508 +0000 UTC m=+1708.583770652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4a041dab-85bd-4b08-86dc-9d078ba1cf60-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" (UID: "4a041dab-85bd-4b08-86dc-9d078ba1cf60") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 06 21:41:11.089944 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.089917    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.090159 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.090138    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a041dab-85bd-4b08-86dc-9d078ba1cf60-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.090282 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.090260    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.102301 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.102281    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpnxp\" (UniqueName: \"kubernetes.io/projected/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kube-api-access-cpnxp\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.593146 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.593086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a041dab-85bd-4b08-86dc-9d078ba1cf60-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.595499 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.595467    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a041dab-85bd-4b08-86dc-9d078ba1cf60-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.801721 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.801680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:11.929589 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:11.929446    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"]
Jul 06 21:41:11.932215 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:41:11.932190    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a041dab_85bd_4b08_86dc_9d078ba1cf60.slice/crio-4c5f8f4ff6e3917d3189cbd07b124cd5c429d0a3c8a5f42e6dda0a6a6aa4c36c WatchSource:0}: Error finding container 4c5f8f4ff6e3917d3189cbd07b124cd5c429d0a3c8a5f42e6dda0a6a6aa4c36c: Status 404 returned error can't find the container with id 4c5f8f4ff6e3917d3189cbd07b124cd5c429d0a3c8a5f42e6dda0a6a6aa4c36c
Jul 06 21:41:12.046109 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:12.046079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" event={"ID":"4a041dab-85bd-4b08-86dc-9d078ba1cf60","Type":"ContainerStarted","Data":"f41b84e923447c2850cb9de0953f6862eb98b85acfe1cdf771abe6667bd3caff"}
Jul 06 21:41:12.046294 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:12.046134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" event={"ID":"4a041dab-85bd-4b08-86dc-9d078ba1cf60","Type":"ContainerStarted","Data":"4c5f8f4ff6e3917d3189cbd07b124cd5c429d0a3c8a5f42e6dda0a6a6aa4c36c"}
Jul 06 21:41:21.078322 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:21.078288    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerID="f41b84e923447c2850cb9de0953f6862eb98b85acfe1cdf771abe6667bd3caff" exitCode=0
Jul 06 21:41:21.078690 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:21.078361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" event={"ID":"4a041dab-85bd-4b08-86dc-9d078ba1cf60","Type":"ContainerDied","Data":"f41b84e923447c2850cb9de0953f6862eb98b85acfe1cdf771abe6667bd3caff"}
Jul 06 21:41:22.084425 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:22.084393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" event={"ID":"4a041dab-85bd-4b08-86dc-9d078ba1cf60","Type":"ContainerStarted","Data":"2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e"}
Jul 06 21:41:22.084846 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:22.084453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" event={"ID":"4a041dab-85bd-4b08-86dc-9d078ba1cf60","Type":"ContainerStarted","Data":"d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4"}
Jul 06 21:41:22.084846 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:22.084638    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:22.105301 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:22.105223    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" podStartSLOduration=3.142950933 podStartE2EDuration="12.105205658s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.962254725s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:41:22.103295884 +0000 UTC m=+1719.097395044" watchObservedRunningTime="2026-07-06 21:41:22.105205658 +0000 UTC m=+1719.099304818"
Jul 06 21:41:23.090856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:23.090821    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:23.092323 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:23.092293    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 06 21:41:24.094221 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:24.094182    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 06 21:41:29.098860 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:29.098833    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:29.099407 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:29.099377    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 06 21:41:39.099924 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:39.099897    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:40.933515 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:40.933483    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"]
Jul 06 21:41:40.933931 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:40.933713    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="kserve-container" containerID="cri-o://d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4" gracePeriod=30
Jul 06 21:41:40.933931 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:40.933772    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="kube-rbac-proxy" containerID="cri-o://2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e" gracePeriod=30
Jul 06 21:41:41.157125 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.157096    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerID="2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e" exitCode=2
Jul 06 21:41:41.157331 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.157168    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" event={"ID":"4a041dab-85bd-4b08-86dc-9d078ba1cf60","Type":"ContainerDied","Data":"2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e"}
Jul 06 21:41:41.782142 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.782117    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:41.917011 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.916981    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpnxp\" (UniqueName: \"kubernetes.io/projected/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kube-api-access-cpnxp\") pod \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") "
Jul 06 21:41:41.917011 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.917011    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a041dab-85bd-4b08-86dc-9d078ba1cf60-proxy-tls\") pod \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") "
Jul 06 21:41:41.917216 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.917052    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kserve-provision-location\") pod \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") "
Jul 06 21:41:41.917216 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.917072    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kube-rbac-proxy-sar-config\") pod \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") "
Jul 06 21:41:41.917404 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.917205    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a041dab-85bd-4b08-86dc-9d078ba1cf60-cabundle-cert\") pod \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\" (UID: \"4a041dab-85bd-4b08-86dc-9d078ba1cf60\") "
Jul 06 21:41:41.917456 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.917398    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a041dab-85bd-4b08-86dc-9d078ba1cf60" (UID: "4a041dab-85bd-4b08-86dc-9d078ba1cf60"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:41:41.917574 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.917554    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a041dab-85bd-4b08-86dc-9d078ba1cf60-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a041dab-85bd-4b08-86dc-9d078ba1cf60" (UID: "4a041dab-85bd-4b08-86dc-9d078ba1cf60"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:41:41.918390 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.918362    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a041dab-85bd-4b08-86dc-9d078ba1cf60" (UID: "4a041dab-85bd-4b08-86dc-9d078ba1cf60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:41.919062 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.919045    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kube-api-access-cpnxp" (OuterVolumeSpecName: "kube-api-access-cpnxp") pod "4a041dab-85bd-4b08-86dc-9d078ba1cf60" (UID: "4a041dab-85bd-4b08-86dc-9d078ba1cf60"). InnerVolumeSpecName "kube-api-access-cpnxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:41:41.919252 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:41.919214    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a041dab-85bd-4b08-86dc-9d078ba1cf60-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a041dab-85bd-4b08-86dc-9d078ba1cf60" (UID: "4a041dab-85bd-4b08-86dc-9d078ba1cf60"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:41:42.018279 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.018219    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpnxp\" (UniqueName: \"kubernetes.io/projected/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kube-api-access-cpnxp\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:41:42.018279 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.018272    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a041dab-85bd-4b08-86dc-9d078ba1cf60-proxy-tls\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:41:42.018279 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.018285    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kserve-provision-location\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:41:42.018700 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.018297    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a041dab-85bd-4b08-86dc-9d078ba1cf60-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:41:42.018700 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.018309    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a041dab-85bd-4b08-86dc-9d078ba1cf60-cabundle-cert\") on node \"ip-10-0-137-165.ec2.internal\" DevicePath \"\""
Jul 06 21:41:42.162489 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.162455    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerID="d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4" exitCode=0
Jul 06 21:41:42.162655 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.162516    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"
Jul 06 21:41:42.162655 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.162540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" event={"ID":"4a041dab-85bd-4b08-86dc-9d078ba1cf60","Type":"ContainerDied","Data":"d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4"}
Jul 06 21:41:42.162655 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.162605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989" event={"ID":"4a041dab-85bd-4b08-86dc-9d078ba1cf60","Type":"ContainerDied","Data":"4c5f8f4ff6e3917d3189cbd07b124cd5c429d0a3c8a5f42e6dda0a6a6aa4c36c"}
Jul 06 21:41:42.162655 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.162631    2581 scope.go:117] "RemoveContainer" containerID="2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e"
Jul 06 21:41:42.171061 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.171041    2581 scope.go:117] "RemoveContainer" containerID="d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4"
Jul 06 21:41:42.178077 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.178063    2581 scope.go:117] "RemoveContainer" containerID="f41b84e923447c2850cb9de0953f6862eb98b85acfe1cdf771abe6667bd3caff"
Jul 06 21:41:42.184789 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.184727    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"]
Jul 06 21:41:42.186978 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.186961    2581 scope.go:117] "RemoveContainer" containerID="2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e"
Jul 06 21:41:42.187429 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:41:42.187350    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e\": container with ID starting with 2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e not found: ID does not exist" containerID="2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e"
Jul 06 21:41:42.187429 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.187388    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e"} err="failed to get container status \"2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e\": rpc error: code = NotFound desc = could not find container \"2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e\": container with ID starting with 2ef37a7d48408f934b13522888b160db32a3a828fd42978c19a2420bf6c67f9e not found: ID does not exist"
Jul 06 21:41:42.187429 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.187414    2581 scope.go:117] "RemoveContainer" containerID="d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4"
Jul 06 21:41:42.187935 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:41:42.187743    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4\": container with ID starting with d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4 not found: ID does not exist" containerID="d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4"
Jul 06 21:41:42.187935 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.187777    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4"} err="failed to get container status \"d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4\": rpc error: code = NotFound desc = could not find container \"d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4\": container with ID starting with d5808527e03aef55af82cdf19356ccaeb8b54b2600da3bb3022b568529bc33f4 not found: ID does not exist"
Jul 06 21:41:42.187935 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.187809    2581 scope.go:117] "RemoveContainer" containerID="f41b84e923447c2850cb9de0953f6862eb98b85acfe1cdf771abe6667bd3caff"
Jul 06 21:41:42.188206 ip-10-0-137-165 kubenswrapper[2581]: E0706 21:41:42.188173    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41b84e923447c2850cb9de0953f6862eb98b85acfe1cdf771abe6667bd3caff\": container with ID starting with f41b84e923447c2850cb9de0953f6862eb98b85acfe1cdf771abe6667bd3caff not found: ID does not exist" containerID="f41b84e923447c2850cb9de0953f6862eb98b85acfe1cdf771abe6667bd3caff"
Jul 06 21:41:42.188639 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.188214    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41b84e923447c2850cb9de0953f6862eb98b85acfe1cdf771abe6667bd3caff"} err="failed to get container status \"f41b84e923447c2850cb9de0953f6862eb98b85acfe1cdf771abe6667bd3caff\": rpc error: code = NotFound desc = could not find container \"f41b84e923447c2850cb9de0953f6862eb98b85acfe1cdf771abe6667bd3caff\": container with ID starting with f41b84e923447c2850cb9de0953f6862eb98b85acfe1cdf771abe6667bd3caff not found: ID does not exist"
Jul 06 21:41:42.189351 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:42.189332    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-qk989"]
Jul 06 21:41:43.877768 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:41:43.877731    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" path="/var/lib/kubelet/pods/4a041dab-85bd-4b08-86dc-9d078ba1cf60/volumes"
Jul 06 21:42:17.117205 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:17.117176    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l4qqt_376861d0-1d29-4779-9269-e87fdc65094a/global-pull-secret-syncer/0.log"
Jul 06 21:42:17.123968 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:17.123949    2581 ???:1] "http: TLS handshake error from 10.0.132.134:58252: EOF"
Jul 06 21:42:17.311110 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:17.311072    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x2nkj_69673591-604c-4157-9715-fcd796eaf2a8/konnectivity-agent/0.log"
Jul 06 21:42:17.384161 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:17.384050    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-165.ec2.internal_2fa6cec54f4cfb0fa895e12eff069ff2/haproxy/0.log"
Jul 06 21:42:22.498371 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:22.498342    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpqt9_7b673fdb-9f97-4c99-ac99-aef2386ef9ed/node-exporter/0.log"
Jul 06 21:42:22.524009 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:22.523978    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpqt9_7b673fdb-9f97-4c99-ac99-aef2386ef9ed/kube-rbac-proxy/0.log"
Jul 06 21:42:22.550803 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:22.550772    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpqt9_7b673fdb-9f97-4c99-ac99-aef2386ef9ed/init-textfile/0.log"
Jul 06 21:42:25.553705 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.553668    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"]
Jul 06 21:42:25.554057 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.553919    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="kserve-container"
Jul 06 21:42:25.554057 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.553929    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="kserve-container"
Jul 06 21:42:25.554057 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.553947    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="storage-initializer"
Jul 06 21:42:25.554057 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.553953    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="storage-initializer"
Jul 06 21:42:25.554057 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.553963    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="kube-rbac-proxy"
Jul 06 21:42:25.554057 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.553969    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="kube-rbac-proxy"
Jul 06 21:42:25.554057 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.554010    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="kserve-container"
Jul 06 21:42:25.554057 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.554018    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a041dab-85bd-4b08-86dc-9d078ba1cf60" containerName="kube-rbac-proxy"
Jul 06 21:42:25.556828 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.556810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.560024 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.560004    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5rcx5\"/\"kube-root-ca.crt\""
Jul 06 21:42:25.560157 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.560066    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5rcx5\"/\"openshift-service-ca.crt\""
Jul 06 21:42:25.560865 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.560851    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5rcx5\"/\"default-dockercfg-9b8k8\""
Jul 06 21:42:25.566547 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.566523    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"]
Jul 06 21:42:25.637503 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.637464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f59c902f-f97e-46b3-bd60-2288550b4f20-sys\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.637676 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.637513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f59c902f-f97e-46b3-bd60-2288550b4f20-lib-modules\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.637676 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.637593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f59c902f-f97e-46b3-bd60-2288550b4f20-podres\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.637676 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.637650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkj2\" (UniqueName: \"kubernetes.io/projected/f59c902f-f97e-46b3-bd60-2288550b4f20-kube-api-access-mtkj2\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.637780 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.637678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f59c902f-f97e-46b3-bd60-2288550b4f20-proc\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.738521 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.738485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f59c902f-f97e-46b3-bd60-2288550b4f20-sys\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.738695 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.738531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f59c902f-f97e-46b3-bd60-2288550b4f20-lib-modules\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.738695 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.738564    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f59c902f-f97e-46b3-bd60-2288550b4f20-podres\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.738695 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.738595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkj2\" (UniqueName: \"kubernetes.io/projected/f59c902f-f97e-46b3-bd60-2288550b4f20-kube-api-access-mtkj2\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.738695 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.738616    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f59c902f-f97e-46b3-bd60-2288550b4f20-sys\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.738695 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.738629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f59c902f-f97e-46b3-bd60-2288550b4f20-proc\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.738864 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.738691    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f59c902f-f97e-46b3-bd60-2288550b4f20-lib-modules\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.738864 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.738704    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f59c902f-f97e-46b3-bd60-2288550b4f20-proc\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.738864 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.738722    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f59c902f-f97e-46b3-bd60-2288550b4f20-podres\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.748045 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.748017    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkj2\" (UniqueName: \"kubernetes.io/projected/f59c902f-f97e-46b3-bd60-2288550b4f20-kube-api-access-mtkj2\") pod \"perf-node-gather-daemonset-bbc68\" (UID: \"f59c902f-f97e-46b3-bd60-2288550b4f20\") " pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.866526 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.866421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:25.998854 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:25.998823    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"]
Jul 06 21:42:26.001981 ip-10-0-137-165 kubenswrapper[2581]: W0706 21:42:26.001955    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf59c902f_f97e_46b3_bd60_2288550b4f20.slice/crio-2725936f319aa4915b71a6265873df21e4f219aeb53d3ce984aa683b5baa4131 WatchSource:0}: Error finding container 2725936f319aa4915b71a6265873df21e4f219aeb53d3ce984aa683b5baa4131: Status 404 returned error can't find the container with id 2725936f319aa4915b71a6265873df21e4f219aeb53d3ce984aa683b5baa4131
Jul 06 21:42:26.316674 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:26.316636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68" event={"ID":"f59c902f-f97e-46b3-bd60-2288550b4f20","Type":"ContainerStarted","Data":"5cba835879f1b4acd30e6020f31acb98cffe875362fdb4fd4a8d764be82691f8"}
Jul 06 21:42:26.316856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:26.316696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68" event={"ID":"f59c902f-f97e-46b3-bd60-2288550b4f20","Type":"ContainerStarted","Data":"2725936f319aa4915b71a6265873df21e4f219aeb53d3ce984aa683b5baa4131"}
Jul 06 21:42:26.316856 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:26.316750    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:26.334752 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:26.334701    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68" podStartSLOduration=1.334686403 podStartE2EDuration="1.334686403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:42:26.333325022 +0000 UTC m=+1783.327424193" watchObservedRunningTime="2026-07-06 21:42:26.334686403 +0000 UTC m=+1783.328785564"
Jul 06 21:42:27.043891 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:27.043857    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9gs9l_8a932825-f754-47ac-80b8-9d1e9a0b7d47/dns/0.log"
Jul 06 21:42:27.067484 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:27.067459    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9gs9l_8a932825-f754-47ac-80b8-9d1e9a0b7d47/kube-rbac-proxy/0.log"
Jul 06 21:42:27.255649 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:27.255620    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vnjbh_c8b1bafa-cee5-4b58-8da4-753d6c0805bd/dns-node-resolver/0.log"
Jul 06 21:42:27.798711 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:27.798679    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nrrrm_25fcb51b-8a18-4ba8-b525-86c727dbabf4/node-ca/0.log"
Jul 06 21:42:29.057835 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:29.057805    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jbvdd_5c08ff3c-4b2c-46c2-bd19-48069125d8d3/serve-healthcheck-canary/0.log"
Jul 06 21:42:29.535164 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:29.535135    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rpsbv_0339a918-5080-42bb-acf8-c610c7db7ca6/kube-rbac-proxy/0.log"
Jul 06 21:42:29.556240 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:29.556215    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rpsbv_0339a918-5080-42bb-acf8-c610c7db7ca6/exporter/0.log"
Jul 06 21:42:29.578619 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:29.578596    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rpsbv_0339a918-5080-42bb-acf8-c610c7db7ca6/extractor/0.log"
Jul 06 21:42:31.777801 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:31.777768    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7ccfc88547-c7vz8_d315926c-34e4-4994-84bc-d80cf47ebcf9/manager/0.log"
Jul 06 21:42:31.833452 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:31.833425    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-67p6q_56eed0bb-9b6d-4fc7-990a-7111497515f3/server/0.log"
Jul 06 21:42:31.933249 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:31.933202    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-mxfcs_75978d84-5ef5-49f6-b486-ce2bb5826b7c/manager/0.log"
Jul 06 21:42:32.118948 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:32.118864    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-qx9x4_6772628d-9c5a-42af-a34c-088a2a3503fb/seaweedfs/0.log"
Jul 06 21:42:32.174678 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:32.174650    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-f8xfx_7a8504e5-800b-429c-ba06-880a996e40ae/seaweedfs-tls-serving/0.log"
Jul 06 21:42:32.329902 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:32.329879    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5rcx5/perf-node-gather-daemonset-bbc68"
Jul 06 21:42:37.929287 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:37.929250    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5thzq_831d79b5-c28f-4857-8b33-0b013f995cea/kube-multus/0.log"
Jul 06 21:42:38.301443 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:38.301369    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s5lgg_dfa5cd09-70f6-4a62-965e-647b129b69be/kube-multus-additional-cni-plugins/0.log"
Jul 06 21:42:38.323896 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:38.323867    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s5lgg_dfa5cd09-70f6-4a62-965e-647b129b69be/egress-router-binary-copy/0.log"
Jul 06 21:42:38.349854 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:38.349830    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s5lgg_dfa5cd09-70f6-4a62-965e-647b129b69be/cni-plugins/0.log"
Jul 06 21:42:38.379210 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:38.379181    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s5lgg_dfa5cd09-70f6-4a62-965e-647b129b69be/bond-cni-plugin/0.log"
Jul 06 21:42:38.403317 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:38.403294    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s5lgg_dfa5cd09-70f6-4a62-965e-647b129b69be/routeoverride-cni/0.log"
Jul 06 21:42:38.426205 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:38.426159    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s5lgg_dfa5cd09-70f6-4a62-965e-647b129b69be/whereabouts-cni-bincopy/0.log"
Jul 06 21:42:38.450459 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:38.450433    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s5lgg_dfa5cd09-70f6-4a62-965e-647b129b69be/whereabouts-cni/0.log"
Jul 06 21:42:38.477649 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:38.477618    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7l279_75a6daab-e23d-43e9-ab04-d52574901b8e/network-metrics-daemon/0.log"
Jul 06 21:42:38.499537 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:38.499507    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7l279_75a6daab-e23d-43e9-ab04-d52574901b8e/kube-rbac-proxy/0.log"
Jul 06 21:42:39.864909 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:39.864878    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6p6t_cc05b91f-f482-4025-b0f3-d9bff11999d2/ovn-controller/0.log"
Jul 06 21:42:39.882566 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:39.882535    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6p6t_cc05b91f-f482-4025-b0f3-d9bff11999d2/ovn-acl-logging/0.log"
Jul 06 21:42:39.914115 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:39.914084    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6p6t_cc05b91f-f482-4025-b0f3-d9bff11999d2/kube-rbac-proxy-node/0.log"
Jul 06 21:42:39.941574 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:39.941551    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6p6t_cc05b91f-f482-4025-b0f3-d9bff11999d2/kube-rbac-proxy-ovn-metrics/0.log"
Jul 06 21:42:39.959866 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:39.959838    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6p6t_cc05b91f-f482-4025-b0f3-d9bff11999d2/northd/0.log"
Jul 06 21:42:39.981242 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:39.981215    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6p6t_cc05b91f-f482-4025-b0f3-d9bff11999d2/nbdb/0.log"
Jul 06 21:42:40.003536 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:40.003503    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6p6t_cc05b91f-f482-4025-b0f3-d9bff11999d2/sbdb/0.log"
Jul 06 21:42:40.064960 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:40.064926    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6p6t_cc05b91f-f482-4025-b0f3-d9bff11999d2/ovnkube-controller/0.log"
Jul 06 21:42:41.155904 ip-10-0-137-165 kubenswrapper[2581]: I0706 21:42:41.155861    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-klq8n_3f33c872-374e-4583-89da-6457f198a950/network-check-target-container/0.log"