Jul 16 21:48:12.377011 ip-10-0-137-85 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 16 21:48:12.377023 ip-10-0-137-85 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 16 21:48:12.377030 ip-10-0-137-85 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 16 21:48:12.377247 ip-10-0-137-85 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 16 21:48:22.484903 ip-10-0-137-85 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 16 21:48:22.484923 ip-10-0-137-85 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4e64325221904a2f8bd123296423052c --
Jul 16 21:50:15.313337 ip-10-0-137-85 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 21:50:15.608109 ip-10-0-137-85 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:50:15.608109 ip-10-0-137-85 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 21:50:15.608109 ip-10-0-137-85 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:50:15.608109 ip-10-0-137-85 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 21:50:15.608109 ip-10-0-137-85 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:50:15.608748 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.608673    2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 21:50:15.612766 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612748    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:15.612766 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612762    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:15.612766 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612766    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:15.612766 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612769    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:15.612766 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612772    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612775    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612778    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612781    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612784    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612787    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612791    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612793    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612796    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612798    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612801    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612804    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612806    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612808    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612811    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612813    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612816    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612818    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612821    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612823    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:15.612943 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612833    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612835    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612838    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612840    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612843    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612846    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612848    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612850    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612853    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612855    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612858    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612860    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612863    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612865    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612870    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612872    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612875    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612879    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612882    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612885    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:15.613421 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612888    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612891    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612893    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612896    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612899    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612901    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612904    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612907    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612909    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612912    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612914    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612918    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612921    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612923    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612925    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612928    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612930    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612933    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612935    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612938    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:15.613872 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612940    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612943    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612946    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612948    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612951    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612953    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612957    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612959    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612961    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612965    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612969    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612972    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612975    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612977    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612980    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612982    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612985    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612987    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612990    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:15.614345 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612993    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612995    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.612998    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613000    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613002    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613005    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613008    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613012    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613014    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613017    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613020    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613023    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613025    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613028    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613031    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613034    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613692    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613699    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613702    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:15.614782 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613706    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613709    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613712    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613714    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613717    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613719    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613722    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613725    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613727    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613730    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613732    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613735    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613737    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613740    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613742    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613745    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613747    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613750    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613752    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613755    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:15.615282 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613757    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613759    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613762    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613764    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613767    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613770    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613772    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613775    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613777    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613780    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613782    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613785    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613787    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613790    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613792    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613795    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613797    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613799    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613802    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613805    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:15.615774 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613808    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613810    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613813    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613815    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613817    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613819    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613822    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613825    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613827    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613829    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613832    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613834    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613836    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613839    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613843    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613845    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613848    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613850    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613853    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:15.616288 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613855    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613857    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613860    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613862    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613865    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613867    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613870    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613873    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613875    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613878    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613880    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613883    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613886    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613889    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613891    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613894    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613896    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613898    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613903    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613906    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:15.616737 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613909    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613912    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613914    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613917    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613921    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613924    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613926    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613929    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613932    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613934    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613937    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613939    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613941    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613944    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613946    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613948    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.613951    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614295    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614300    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:15.617216 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614303    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614306    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614308    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614311    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614313    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614316    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614319    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614321    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614324    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614326    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614329    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614331    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614333    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614336    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614338    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614341    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614344    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614346    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614349    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614352    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:15.617691 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614355    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614357    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614360    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614362    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614365    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614367    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614370    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614372    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614375    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614378    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614381    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614383    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614386    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614388    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614390    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614393    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614395    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614398    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614400    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:15.618174 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614403    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614405    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614408    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614411    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614413    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614416    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614421    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614425    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614429    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614432    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614435    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614437    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614440    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614442    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614445    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614448    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614450    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614453    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614455    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:15.618607 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614457    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614460    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614462    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614465    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614467    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614470    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614472    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614474    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614477    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614479    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614481    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614485    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614487    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614489    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614492    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614494    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614497    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614499    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614501    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614504    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:15.619241 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614506    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614509    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614511    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614513    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614516    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614518    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614520    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614523    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614525    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614528    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614530    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614532    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614534    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614537    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614540    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614543    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614545    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614548    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.614550    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614616    2570 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614622    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 21:50:15.619721 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614627    2570 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614633    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614638    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614643    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614648    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614654    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614659    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614664    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614667    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614670    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614673    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614676    2570 flags.go:64] FLAG: --cgroup-root=""
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614680    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614683    2570 flags.go:64] FLAG: --client-ca-file=""
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614685    2570 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614688    2570 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614693    2570 flags.go:64] FLAG: --cluster-domain=""
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614695    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614698    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614701    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614704    2570 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614708    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614710    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 21:50:15.620236 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614713    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614716    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614719    2570 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614721    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614724    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614727    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614730    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614733    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614736    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614739    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614742    2570 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614744    2570 flags.go:64] FLAG: --enable-server="true"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614747    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614750    2570 flags.go:64] FLAG: --event-burst="100"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614753    2570 flags.go:64] FLAG: --event-qps="50"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614756    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614759    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614762    2570 flags.go:64] FLAG: --eviction-hard=""
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614765    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614768    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614771    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614775    2570 flags.go:64] FLAG: --eviction-soft=""
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614778    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614781    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614783    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 21:50:15.620912 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614786    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614789    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614792    2570 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614795    2570 flags.go:64] FLAG: --feature-gates=""
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614799    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614802    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614804    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614807    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614810    2570 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614813    2570 flags.go:64] FLAG: --help="false"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614816    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-137-85.ec2.internal"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614818    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614821    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614824    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614827    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614830    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614833    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614835    2570 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614838    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614841    2570 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614843    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614846    2570 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614849    2570 flags.go:64] FLAG: --kube-reserved=""
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614851    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 21:50:15.621525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614854    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614857    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614860    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614863    2570 flags.go:64] FLAG: --lock-file=""
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614867    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614869    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614872    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614877    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614880    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614882    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614886    2570 flags.go:64] FLAG: --logging-format="text"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614889    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614893    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614895    2570 flags.go:64] FLAG: --manifest-url=""
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614898    2570 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614902    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614904    2570 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614908    2570 flags.go:64] FLAG: --max-pods="110"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614911    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614913    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614916    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614919    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614921    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614924    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614927    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 21:50:15.622101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614933    2570 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614936    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614939    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614942    2570 flags.go:64] FLAG: --pod-cidr=""
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614945    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614949    2570 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614951    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614954    2570 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614957    2570 flags.go:64] FLAG: --port="10250"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614960    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614963    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a097bee176ebb8ef"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614966    2570 flags.go:64] FLAG: --qos-reserved=""
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614970    2570 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614973    2570 flags.go:64] FLAG: --register-node="true"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614976    2570 flags.go:64] FLAG: --register-with-taints=""
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614979    2570 flags.go:64] FLAG: --registry-burst="10"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614982    2570 flags.go:64] FLAG: --registry-qps="5"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614986    2570 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614989    2570 flags.go:64] FLAG: --reserved-memory=""
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614992    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614995    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.614997    2570 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615008    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615012    2570 flags.go:64] FLAG: --runonce="false"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615014    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 21:50:15.622690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615017    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615020    2570 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615023    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615025    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615028    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615031    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615034    2570 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615054    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615057    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615060    2570 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615062    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615065    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615068    2570 flags.go:64] FLAG: --system-cgroups=""
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615071    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615076    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615079    2570 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615093    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615098    2570 flags.go:64] FLAG: --tls-min-version=""
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615101    2570 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615105    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615108    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615110    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615114    2570 flags.go:64] FLAG: --v="2"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615117    2570 flags.go:64] FLAG: --version="false"
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615122    2570 flags.go:64] FLAG: --vmodule=""
Jul 16 21:50:15.623325 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615126    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.615129    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615247    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615251    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615254    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615257    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615260    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615262    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615265    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615268    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615270    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615273    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615275    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615278    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615280    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615283    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615286    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615289    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615292    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615294    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:15.623951 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615297    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615299    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615302    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615305    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615309    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615311    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615315    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615318    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615320    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615323    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615325    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615330    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615333    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615335    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615339    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615343    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615346    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615349    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615352    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:15.624452 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615355    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615357    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615360    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615363    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615365    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615368    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615372    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615375    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615378    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615381    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615384    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615386    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615389    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615391    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615394    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615396    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615399    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615401    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615404    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:15.624889 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615408    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615410    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615413    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615415    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615418    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615422    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615425    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615427    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615430    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615432    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615435    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615437    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615440    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615442    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615445    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615447    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615450    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615452    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615455    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615457    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:15.625346 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615460    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615462    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615465    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615467    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615470    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615473    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615475    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615477    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615480    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615482    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615485    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615487    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615492    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615494    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615497    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615501    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615504    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615509    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615511    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:15.625812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615514    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615517    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615519    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.615522    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.616027    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.621686    2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.621702    2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.621720    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.621726    2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621775    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621780    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621783    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621786    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621789    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:15.626275 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621791    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621794    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621797    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621800    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621802    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621806    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621809    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621811    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621814    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621817    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621819    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621822    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621824    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621827    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621830    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621834    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621837    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621840    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621843    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621846    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:15.626608 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621848    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621851    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621854    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621856    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621860    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621863    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621865    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621868    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621871    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621873    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621876    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621878    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621881    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621883    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621886    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621888    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621890    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621893    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621895    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:15.627115 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621898    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621900    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621903    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621905    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621907    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621910    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621912    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621914    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621917    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621920    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621922    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621925    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621927    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621929    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621932    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621934    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621936    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621940    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621943    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621946    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:15.627576 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621948    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621950    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621953    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621955    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621957    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621960    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621962    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621965    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621967    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621969    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621973    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621977    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621979    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621982    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621985    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621988    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621991    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621994    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621996    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:15.628063 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.621999    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622002    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622004    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622006    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622009    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622011    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622014    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622017    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622019    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622022    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622025    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622027    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622030    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622032    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622036    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:15.628551 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622039    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.622044    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622162    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622168    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622171    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622173    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622176    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622178    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622181    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622183    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622186    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622188    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622191    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622193    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622195    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:15.628897 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622198    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622200    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622203    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622205    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622208    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622210    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622213    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622217    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622220    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622223    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622225    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622227    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622230    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622233    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622235    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622238    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622240    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622243    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622245    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622247    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:15.629299 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622250    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622252    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622254    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622257    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622259    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622261    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622264    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622266    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622269    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622271    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622273    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622276    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622278    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622280    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622283    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622285    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622288    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622290    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622293    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622295    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:15.629755 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622297    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622300    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622302    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622305    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622308    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622310    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622312    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622315    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622317    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622320    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622322    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622325    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622327    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622330    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622332    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622335    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622337    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622339    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622343    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:15.630233 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622346    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622349    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622351    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622354    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622356    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622359    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622361    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622364    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622366    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622369    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622371    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622373    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622376    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622378    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622380    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622384    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622387    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622390    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622393    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622396    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:15.630684 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622398    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:15.631183 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622401    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:15.631183 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622404    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:15.631183 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622407    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:15.631183 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622410    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:15.631183 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622412    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:15.631183 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:15.622415    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:15.631183 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.622419    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:50:15.631183 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.622511    2570 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 21:50:15.690429 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.690414    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 21:50:15.693534 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.693517    2570 server.go:1019] "Starting client certificate rotation"
Jul 16 21:50:15.693649 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.693631    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:50:15.693719 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.693680    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:50:15.695435 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.695420    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:50:15.696541 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.696526    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:50:15.704585 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.704563    2570 log.go:25] "Validated CRI v1 runtime API"
Jul 16 21:50:15.706410 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.706396    2570 log.go:25] "Validated CRI v1 image API"
Jul 16 21:50:15.707256 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.707238    2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 21:50:15.709286 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.709270    2570 fs.go:135] Filesystem UUIDs: map[2e6bf893-a272-48f0-892a-2e90f4b29d35:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 86ed1292-8cae-4e94-84a4-7403e383d3f2:/dev/nvme0n1p4]
Jul 16 21:50:15.709344 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.709286    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 21:50:15.714882 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.714739    2570 manager.go:216] Machine: {Timestamp:2026-07-16 21:50:15.713165752 +0000 UTC m=+0.310705974 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3138713 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b070ad5ccfea1d9bdf97985e0b36b SystemUUID:ec2b070a-d5cc-fea1-d9bd-f97985e0b36b BootID:4e643252-2190-4a2f-8bd1-23296423052c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:db:89:b3:c3:8f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:db:89:b3:c3:8f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:4a:87:24:8a:3f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 21:50:15.714882 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.714877    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 21:50:15.714982 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.714947    2570 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 21:50:15.717249 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.717221    2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 21:50:15.717396 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.717252    2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-85.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 21:50:15.717877 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.717866    2570 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 21:50:15.717918 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.717884    2570 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 21:50:15.717968 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.717921    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:50:15.718002 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.717922    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:50:15.718475 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.718464    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:50:15.719534 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.719522    2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 21:50:15.723078 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.723064    2570 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:50:15.723219 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.723208    2570 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 21:50:15.723275 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.723243    2570 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 21:50:15.723275 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.723256    2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 21:50:15.723275 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.723274    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 21:50:15.723401 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.723285    2570 kubelet.go:398] "Adding apiserver pod source"
Jul 16 21:50:15.723401 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.723299    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 21:50:15.724152 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.724130    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:50:15.724235 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.724160    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:50:15.725925 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.725901    2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 21:50:15.727572 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.727555    2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 21:50:15.727650 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.727594    2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 21:50:15.728615 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.728601    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 21:50:15.728680 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.728620    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 21:50:15.728680 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.728628    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 21:50:15.728680 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.728635    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 21:50:15.728680 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.728646    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 21:50:15.728680 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.728654    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 21:50:15.728680 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.728665    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 21:50:15.728680 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.728672    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 21:50:15.728680 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.728681    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 21:50:15.728928 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.728689    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 21:50:15.728928 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.728701    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 21:50:15.728928 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.728709    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 21:50:15.729299 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.729288    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 21:50:15.729347 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.729302    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 21:50:15.732120 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.732106    2570 server.go:1268] "Started kubelet"
Jul 16 21:50:15.732227 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.732187    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 21:50:15.732323 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.732198    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 21:50:15.732323 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.732274    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 21:50:15.732668 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.732642    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-85.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 21:50:15.732668 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.732642    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 21:50:15.732800 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.732694    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-85.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 21:50:15.732830 ip-10-0-137-85 systemd[1]: Started Kubernetes Kubelet.
Jul 16 21:50:15.734059 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.734047    2570 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 21:50:15.736216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.736201    2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 21:50:15.736773 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.736754    2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 21:50:15.737660 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.736977    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-85.ec2.internal.18c2e391553e9260  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-85.ec2.internal,UID:ip-10-0-137-85.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-85.ec2.internal,},FirstTimestamp:2026-07-16 21:50:15.732064864 +0000 UTC m=+0.329605088,LastTimestamp:2026-07-16 21:50:15.732064864 +0000 UTC m=+0.329605088,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-85.ec2.internal,}"
Jul 16 21:50:15.741222 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.741203    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 21:50:15.741301 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.741220    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 21:50:15.741301 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.741285    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:15.741714 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.741687    2570 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 21:50:15.741714 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.741706    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 21:50:15.741836 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.741724    2570 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 21:50:15.741882 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.741855    2570 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 21:50:15.741882 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.741865    2570 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 21:50:15.742069 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.742053    2570 factory.go:153] Registering CRI-O factory
Jul 16 21:50:15.742167 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.742108    2570 factory.go:223] Registration of the crio container factory successfully
Jul 16 21:50:15.742167 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.742157    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 21:50:15.742167 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.742167    2570 factory.go:55] Registering systemd factory
Jul 16 21:50:15.742318 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.742175    2570 factory.go:223] Registration of the systemd container factory successfully
Jul 16 21:50:15.742318 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.742195    2570 factory.go:103] Registering Raw factory
Jul 16 21:50:15.742318 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.742208    2570 manager.go:1196] Started watching for new ooms in manager
Jul 16 21:50:15.742551 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.742540    2570 manager.go:349] Starting recovery of all containers
Jul 16 21:50:15.746421 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.746394    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xlmdt"
Jul 16 21:50:15.753563 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.753179    2570 manager.go:354] Recovery completed
Jul 16 21:50:15.754644 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.754627    2570 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 16 21:50:15.755607 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.755584    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 21:50:15.755752 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.755733    2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-85.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 21:50:15.757791 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.757780    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:15.759882 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.759866    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:15.759932 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.759899    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:15.759932 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.759914    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:15.760494 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.760481    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 21:50:15.760547 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.760494    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:15.760547 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.760503    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 21:50:15.760547 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.760518    2570 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:50:15.761128 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.761114    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:15.761173 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.761142    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:15.761173 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.761154    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:15.762004 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.761949    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-85.ec2.internal.18c2e39156e70aeb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-85.ec2.internal,UID:ip-10-0-137-85.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-85.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-85.ec2.internal,},FirstTimestamp:2026-07-16 21:50:15.759882987 +0000 UTC m=+0.357423212,LastTimestamp:2026-07-16 21:50:15.759882987 +0000 UTC m=+0.357423212,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-85.ec2.internal,}"
Jul 16 21:50:15.762631 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.762620    2570 policy_none.go:49] "None policy: Start"
Jul 16 21:50:15.762664 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.762636    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 21:50:15.762664 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.762645    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 21:50:15.764528 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.764518    2570 policy_none.go:47] "Start"
Jul 16 21:50:15.769628 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.769529    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xlmdt"
Jul 16 21:50:15.780156 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.780062    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-85.ec2.internal.18c2e39156e76763  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-85.ec2.internal,UID:ip-10-0-137-85.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-85.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-85.ec2.internal,},FirstTimestamp:2026-07-16 21:50:15.759906659 +0000 UTC m=+0.357446885,LastTimestamp:2026-07-16 21:50:15.759906659 +0000 UTC m=+0.357446885,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-85.ec2.internal,}"
Jul 16 21:50:15.806562 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.806547    2570 manager.go:337] "Starting Device Plugin manager"
Jul 16 21:50:15.806626 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.806581    2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 21:50:15.806626 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.806593    2570 server.go:87] "Starting device plugin registration server"
Jul 16 21:50:15.806780 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.806770    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 21:50:15.806811 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.806781    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 21:50:15.806887 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.806874    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 21:50:15.806962 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.806949    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 21:50:15.806962 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.806960    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 21:50:15.807454 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.807437    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 21:50:15.807506 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.807477    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:15.864851 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.864808    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 21:50:15.867142 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.865875    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 21:50:15.867142 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.865905    2570 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 21:50:15.867142 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.865924    2570 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 21:50:15.867142 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.865952    2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 21:50:15.867974 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.867957    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:15.908195 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.908175    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:15.908940 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.908928    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:15.908997 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.908957    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:15.908997 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.908967    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:15.909053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.908998    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-85.ec2.internal"
Jul 16 21:50:15.923187 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.923170    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-85.ec2.internal"
Jul 16 21:50:15.923240 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.923189    2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-85.ec2.internal\": node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:15.949142 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.949108    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:15.966109 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.966069    2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-85.ec2.internal"]
Jul 16 21:50:15.966159 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.966139    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:15.966876 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.966863    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:15.966935 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.966886    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:15.966935 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.966896    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:15.969292 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.969280    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:15.969407 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.969391    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:15.969459 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.969425    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:15.969956 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.969935    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:15.969956 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.969958    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:15.970114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.969970    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:15.970114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.969987    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:15.970114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.970006    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:15.970114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.970014    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:15.972258 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.972245    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:15.972300 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.972267    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:15.972921 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.972902    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:15.972993 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.972934    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:15.972993 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:15.972944    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:15.997651 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:15.997629    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-85.ec2.internal\" not found" node="ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.001832 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:16.001815    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-85.ec2.internal\" not found" node="ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.043975 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.043954    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d67fa00a9f25b2a9762a6da7434b1b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal\" (UID: \"0d67fa00a9f25b2a9762a6da7434b1b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.044035 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.043976    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d67fa00a9f25b2a9762a6da7434b1b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal\" (UID: \"0d67fa00a9f25b2a9762a6da7434b1b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.044035 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.044017    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/186a905c8a5ab6ff537de73d191d65e4-config\") pod \"kube-apiserver-proxy-ip-10-0-137-85.ec2.internal\" (UID: \"186a905c8a5ab6ff537de73d191d65e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.049371 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:16.049357    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:16.144634 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.144591    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d67fa00a9f25b2a9762a6da7434b1b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal\" (UID: \"0d67fa00a9f25b2a9762a6da7434b1b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.144634 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.144613    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d67fa00a9f25b2a9762a6da7434b1b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal\" (UID: \"0d67fa00a9f25b2a9762a6da7434b1b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.144735 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.144640    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/186a905c8a5ab6ff537de73d191d65e4-config\") pod \"kube-apiserver-proxy-ip-10-0-137-85.ec2.internal\" (UID: \"186a905c8a5ab6ff537de73d191d65e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.144735 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.144676    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d67fa00a9f25b2a9762a6da7434b1b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal\" (UID: \"0d67fa00a9f25b2a9762a6da7434b1b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.144735 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.144682    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d67fa00a9f25b2a9762a6da7434b1b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal\" (UID: \"0d67fa00a9f25b2a9762a6da7434b1b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.144735 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.144724    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/186a905c8a5ab6ff537de73d191d65e4-config\") pod \"kube-apiserver-proxy-ip-10-0-137-85.ec2.internal\" (UID: \"186a905c8a5ab6ff537de73d191d65e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.149699 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:16.149685    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:16.250285 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:16.250251    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:16.300403 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.300377    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.304362 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.304317    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:16.350666 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:16.350646    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:16.450903 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:16.450849    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:16.551905 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:16.551877    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:16.652348 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:16.652325    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:16.695764 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.695731    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 21:50:16.695922 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.695898    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:50:16.752921 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:16.752861    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:16.773874 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.773830    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:45:16 +0000 UTC" deadline="2027-12-18 22:02:23.578748544 +0000 UTC"
Jul 16 21:50:16.773874 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.773868    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12480h12m6.80488404s"
Jul 16 21:50:16.846412 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.846385    2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:16.853316 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:16.853297    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:16.869565 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.869546    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:16.870479 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.870460    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:16.870562 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.870497    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:16.870562 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:16.870510    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:16.953892 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:16.953869    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:17.054750 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.054729    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:17.155627 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.155607    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:17.159803 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:17.159773    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186a905c8a5ab6ff537de73d191d65e4.slice/crio-7d6672ba7182c2e34e32ff422d77f9a49b7dcab8b57a95da6c2013d5c2dcb213 WatchSource:0}: Error finding container 7d6672ba7182c2e34e32ff422d77f9a49b7dcab8b57a95da6c2013d5c2dcb213: Status 404 returned error can't find the container with id 7d6672ba7182c2e34e32ff422d77f9a49b7dcab8b57a95da6c2013d5c2dcb213
Jul 16 21:50:17.160379 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:17.160357    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d67fa00a9f25b2a9762a6da7434b1b7.slice/crio-9941e147a774e0a452b5ff8ece1511eda44f8edc9a6d939853cd3b7b04cc4b98 WatchSource:0}: Error finding container 9941e147a774e0a452b5ff8ece1511eda44f8edc9a6d939853cd3b7b04cc4b98: Status 404 returned error can't find the container with id 9941e147a774e0a452b5ff8ece1511eda44f8edc9a6d939853cd3b7b04cc4b98
Jul 16 21:50:17.164366 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.164350    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:50:17.256235 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.256208    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-85.ec2.internal\" not found"
Jul 16 21:50:17.300204 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.300155    2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:17.321961 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.321943    2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:17.342394 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.342371    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:17.355415 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.355399    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:50:17.356259 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.356246    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-85.ec2.internal"
Jul 16 21:50:17.366647 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.366629    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:50:17.725656 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.725600    2570 apiserver.go:52] "Watching apiserver"
Jul 16 21:50:17.736412 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.736389    2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 21:50:17.736862 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.736830    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-m7rl6","kube-system/kube-apiserver-proxy-ip-10-0-137-85.ec2.internal","openshift-cluster-node-tuning-operator/tuned-6g6ph","openshift-image-registry/node-ca-7wss5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal","openshift-multus/multus-additional-cni-plugins-p4gx4","openshift-multus/network-metrics-daemon-c2dgn","openshift-network-diagnostics/network-check-target-s9mhc","kube-system/konnectivity-agent-xn4rj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6","openshift-dns/node-resolver-t54hm","openshift-multus/multus-njxsf","openshift-network-operator/iptables-alerter-rxphl","openshift-ovn-kubernetes/ovnkube-node-r6hvb"]
Jul 16 21:50:17.739519 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.739492    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:17.739641 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.739599    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:17.742738 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.742709    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 21:50:17.742856 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.742785    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t54hm"
Jul 16 21:50:17.745432 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.745172    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.748120 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.747714    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7wss5"
Jul 16 21:50:17.750477 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.750427    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.753044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.752544    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 21:50:17.753044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.752600    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:17.753044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.752547    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 21:50:17.753044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.752655    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:17.753044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.752802    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rrf4x\""
Jul 16 21:50:17.753044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.752889    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s824p\""
Jul 16 21:50:17.753044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.752938    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 21:50:17.753736 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.753135    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7rhgs\""
Jul 16 21:50:17.755200 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.754564    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756122    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-tmp\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756190    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xclrx\" (UniqueName: \"kubernetes.io/projected/b2365d47-8e55-4776-b882-49eead286f8c-kube-api-access-xclrx\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756243    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ae773a5-b7eb-4346-a16f-186c5f0ec2a8-tmp-dir\") pod \"node-resolver-t54hm\" (UID: \"9ae773a5-b7eb-4346-a16f-186c5f0ec2a8\") " pod="openshift-dns/node-resolver-t54hm"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756284    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-sysctl-d\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756332    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-sysctl-conf\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756365    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-systemd\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756399    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-run\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756432    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-sys\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756471    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756550    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-kubernetes\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756746    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-lib-modules\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756949    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.756948    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-host\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.757104    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ncwz\" (UniqueName: \"kubernetes.io/projected/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-kube-api-access-9ncwz\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.757415    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.757545    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.757679    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.757418    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ae773a5-b7eb-4346-a16f-186c5f0ec2a8-hosts-file\") pod \"node-resolver-t54hm\" (UID: \"9ae773a5-b7eb-4346-a16f-186c5f0ec2a8\") " pod="openshift-dns/node-resolver-t54hm"
Jul 16 21:50:17.758133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.757771    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-sysconfig\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.759238 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.757784    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 21:50:17.759238 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.757808    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 21:50:17.759238 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.757819    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w68vb\" (UniqueName: \"kubernetes.io/projected/9ae773a5-b7eb-4346-a16f-186c5f0ec2a8-kube-api-access-w68vb\") pod \"node-resolver-t54hm\" (UID: \"9ae773a5-b7eb-4346-a16f-186c5f0ec2a8\") " pod="openshift-dns/node-resolver-t54hm"
Jul 16 21:50:17.759238 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.757854    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-modprobe-d\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.759238 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.757896    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-var-lib-kubelet\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.759238 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.757930    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-tuned\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.759238 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.758133    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dfk6l\""
Jul 16 21:50:17.759238 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.758488    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 21:50:17.759238 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.758678    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 21:50:17.759682 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.759488    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xn4rj"
Jul 16 21:50:17.763188 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.762865    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxphl"
Jul 16 21:50:17.763889 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.763810    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 21:50:17.763889 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.763830    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 21:50:17.764059 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.763902    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 21:50:17.764059 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.763927    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 21:50:17.764250 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.764229    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 21:50:17.764465 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.764448    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 21:50:17.765199 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.764842    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jsjqp\""
Jul 16 21:50:17.766267 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.766060    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 21:50:17.767124 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.767103    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:17.767221 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.767175    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:17.768344 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.768321    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xzz8h\""
Jul 16 21:50:17.770062 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.770038    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.772629 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.772607    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:17.772728 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.772676    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:17.775338 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.775278    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.786939 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.786806    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:17.787052 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.786983    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:17.787175 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.787148    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 21:50:17.787350 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.787330    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g2c6x\""
Jul 16 21:50:17.789409 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.789387    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 21:50:17.790448 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.790430    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 21:50:17.790634 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.790615    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 21:50:17.790777 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.790757    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 21:50:17.790930 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.790906    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-72hsc\""
Jul 16 21:50:17.792634 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.792436    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4v8lq\""
Jul 16 21:50:17.797274 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.797252    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:50:17.837284 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.837262    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2zpc4"
Jul 16 21:50:17.843958 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.843932    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 21:50:17.847015 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.846991    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2zpc4"
Jul 16 21:50:17.858500 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858150    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65bfdb19-9745-4a5e-842a-05df0b73d7c0-cnibin\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.858500 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-sysconfig\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.858500 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858216    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vptz5\" (UniqueName: \"kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5\") pod \"network-check-target-s9mhc\" (UID: \"d5c93b6e-7ae4-4340-86bd-c10c6b804523\") " pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:17.858500 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858241    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-socket-dir\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.858500 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858272    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.858500 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858295    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.858500 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-modprobe-d\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.858500 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858361    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-var-lib-kubelet\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.858500 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858418    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-var-lib-kubelet\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.858500 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858418    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-device-dir\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.858500 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858418    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-sysconfig\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.858500 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858478    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-cni-binary-copy\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858514    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-etc-kubernetes\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858528    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-modprobe-d\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858569    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-var-lib-openvswitch\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858606    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-sys\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858631    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-run-multus-certs\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858656    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q86pw\" (UniqueName: \"kubernetes.io/projected/65bfdb19-9745-4a5e-842a-05df0b73d7c0-kube-api-access-q86pw\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858715    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-sys\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858744    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbbaa953-ab0b-4594-a449-06fab72528f5-dbus\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858792    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-multus-socket-dir-parent\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858821    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-var-lib-cni-bin\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858866    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4cnj\" (UniqueName: \"kubernetes.io/projected/902dfcac-4523-4a6a-9338-cbe2661ad1ec-kube-api-access-v4cnj\") pod \"node-ca-7wss5\" (UID: \"902dfcac-4523-4a6a-9338-cbe2661ad1ec\") " pod="openshift-image-registry/node-ca-7wss5"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858891    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-slash\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858922    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ae773a5-b7eb-4346-a16f-186c5f0ec2a8-hosts-file\") pod \"node-resolver-t54hm\" (UID: \"9ae773a5-b7eb-4346-a16f-186c5f0ec2a8\") " pod="openshift-dns/node-resolver-t54hm"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858962    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-var-lib-kubelet\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.858985    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/902dfcac-4523-4a6a-9338-cbe2661ad1ec-host\") pod \"node-ca-7wss5\" (UID: \"902dfcac-4523-4a6a-9338-cbe2661ad1ec\") " pod="openshift-image-registry/node-ca-7wss5"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwj6g\" (UniqueName: \"kubernetes.io/projected/93848932-9ec1-45e6-aec9-34f0a623b9ef-kube-api-access-fwj6g\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.859114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859049    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65bfdb19-9745-4a5e-842a-05df0b73d7c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859065    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ae773a5-b7eb-4346-a16f-186c5f0ec2a8-hosts-file\") pod \"node-resolver-t54hm\" (UID: \"9ae773a5-b7eb-4346-a16f-186c5f0ec2a8\") " pod="openshift-dns/node-resolver-t54hm"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859075    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9vdp\" (UniqueName: \"kubernetes.io/projected/8de28ccc-12c5-4f99-acb9-4e35d577d052-kube-api-access-q9vdp\") pod \"iptables-alerter-rxphl\" (UID: \"8de28ccc-12c5-4f99-acb9-4e35d577d052\") " pod="openshift-network-operator/iptables-alerter-rxphl"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859103    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-tuned\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859160    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-tmp\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859190    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-sysctl-d\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859217    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-sysctl-conf\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859242    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-systemd\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859273    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-hostroot\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859317    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-run-netns\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859342    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-etc-openvswitch\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859389    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93848932-9ec1-45e6-aec9-34f0a623b9ef-ovn-node-metrics-cert\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859430    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65bfdb19-9745-4a5e-842a-05df0b73d7c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859468    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859502    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-host\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859527    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859562    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-etc-selinux\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.860089 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859584    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-host\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859597    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-727cj\" (UniqueName: \"kubernetes.io/projected/385e54b7-90e0-441c-8755-be9e7018fca2-kube-api-access-727cj\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859687    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-system-cni-dir\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859715    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-cnibin\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.859750    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859743    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-multus-conf-dir\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.859894    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-systemd\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.859953    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs podName:b2365d47-8e55-4776-b882-49eead286f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:50:18.359900046 +0000 UTC m=+2.957440259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs") pod "network-metrics-daemon-c2dgn" (UID: "b2365d47-8e55-4776-b882-49eead286f8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860035    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-sysctl-d\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860118    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65bfdb19-9745-4a5e-842a-05df0b73d7c0-os-release\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860148    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-registration-dir\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860172    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-systemd-units\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860215    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-node-log\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860263    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65bfdb19-9745-4a5e-842a-05df0b73d7c0-system-cni-dir\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860313    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-sysctl-conf\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860378    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8de28ccc-12c5-4f99-acb9-4e35d577d052-iptables-alerter-script\") pod \"iptables-alerter-rxphl\" (UID: \"8de28ccc-12c5-4f99-acb9-4e35d577d052\") " pod="openshift-network-operator/iptables-alerter-rxphl"
Jul 16 21:50:17.861049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860409    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbbaa953-ab0b-4594-a449-06fab72528f5-kubelet-config\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860448    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68vb\" (UniqueName: \"kubernetes.io/projected/9ae773a5-b7eb-4346-a16f-186c5f0ec2a8-kube-api-access-w68vb\") pod \"node-resolver-t54hm\" (UID: \"9ae773a5-b7eb-4346-a16f-186c5f0ec2a8\") " pod="openshift-dns/node-resolver-t54hm"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860473    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ac12a88-9208-4759-9c96-3e5de06602a3-konnectivity-ca\") pod \"konnectivity-agent-xn4rj\" (UID: \"4ac12a88-9208-4759-9c96-3e5de06602a3\") " pod="kube-system/konnectivity-agent-xn4rj"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860501    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-multus-cni-dir\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860540    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-var-lib-cni-multus\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860666    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xclrx\" (UniqueName: \"kubernetes.io/projected/b2365d47-8e55-4776-b882-49eead286f8c-kube-api-access-xclrx\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860708    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ae773a5-b7eb-4346-a16f-186c5f0ec2a8-tmp-dir\") pod \"node-resolver-t54hm\" (UID: \"9ae773a5-b7eb-4346-a16f-186c5f0ec2a8\") " pod="openshift-dns/node-resolver-t54hm"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860752    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-run-ovn\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860778    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860809    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-cni-netd\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860834    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93848932-9ec1-45e6-aec9-34f0a623b9ef-ovnkube-script-lib\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.860858    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65bfdb19-9745-4a5e-842a-05df0b73d7c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861088    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ae773a5-b7eb-4346-a16f-186c5f0ec2a8-tmp-dir\") pod \"node-resolver-t54hm\" (UID: \"9ae773a5-b7eb-4346-a16f-186c5f0ec2a8\") " pod="openshift-dns/node-resolver-t54hm"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861151    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-lib-modules\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861208    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ncwz\" (UniqueName: \"kubernetes.io/projected/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-kube-api-access-9ncwz\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861241    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-log-socket\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861290    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-cni-bin\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.861885 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861318    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93848932-9ec1-45e6-aec9-34f0a623b9ef-env-overrides\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861350    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-lib-modules\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861358    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65bfdb19-9745-4a5e-842a-05df0b73d7c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861419    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ac12a88-9208-4759-9c96-3e5de06602a3-agent-certs\") pod \"konnectivity-agent-xn4rj\" (UID: \"4ac12a88-9208-4759-9c96-3e5de06602a3\") " pod="kube-system/konnectivity-agent-xn4rj"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861444    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-os-release\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861469    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-run-netns\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861492    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/902dfcac-4523-4a6a-9338-cbe2661ad1ec-serviceca\") pod \"node-ca-7wss5\" (UID: \"902dfcac-4523-4a6a-9338-cbe2661ad1ec\") " pod="openshift-image-registry/node-ca-7wss5"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861517    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861568    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-kubelet\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861597    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-run-openvswitch\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861631    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-run\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861664    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-sys-fs\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861688    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-run-k8s-cni-cncf-io\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861711    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-multus-daemon-config\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861738    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93848932-9ec1-45e6-aec9-34f0a623b9ef-ovnkube-config\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861776    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8de28ccc-12c5-4f99-acb9-4e35d577d052-host-slash\") pod \"iptables-alerter-rxphl\" (UID: \"8de28ccc-12c5-4f99-acb9-4e35d577d052\") " pod="openshift-network-operator/iptables-alerter-rxphl"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861799    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-kubernetes\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.862720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861833    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twhl4\" (UniqueName: \"kubernetes.io/projected/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-kube-api-access-twhl4\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.863534 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.861863    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-run-systemd\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.863534 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.862023    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-run\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.863534 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.862100    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-kubernetes\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.863534 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.863220    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-etc-tuned\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.863534 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.863288    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-tmp\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.872301 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.872256    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-85.ec2.internal" event={"ID":"186a905c8a5ab6ff537de73d191d65e4","Type":"ContainerStarted","Data":"7d6672ba7182c2e34e32ff422d77f9a49b7dcab8b57a95da6c2013d5c2dcb213"}
Jul 16 21:50:17.874109 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.874077    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal" event={"ID":"0d67fa00a9f25b2a9762a6da7434b1b7","Type":"ContainerStarted","Data":"9941e147a774e0a452b5ff8ece1511eda44f8edc9a6d939853cd3b7b04cc4b98"}
Jul 16 21:50:17.878788 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.876440    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xclrx\" (UniqueName: \"kubernetes.io/projected/b2365d47-8e55-4776-b882-49eead286f8c-kube-api-access-xclrx\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:17.878788 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.877533    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ncwz\" (UniqueName: \"kubernetes.io/projected/d9bb9d45-6b80-4455-9c4c-0cab45a12c5f-kube-api-access-9ncwz\") pod \"tuned-6g6ph\" (UID: \"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:17.878788 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.877884    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w68vb\" (UniqueName: \"kubernetes.io/projected/9ae773a5-b7eb-4346-a16f-186c5f0ec2a8-kube-api-access-w68vb\") pod \"node-resolver-t54hm\" (UID: \"9ae773a5-b7eb-4346-a16f-186c5f0ec2a8\") " pod="openshift-dns/node-resolver-t54hm"
Jul 16 21:50:17.962153 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962120    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-multus-socket-dir-parent\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.962277 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962164    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-var-lib-cni-bin\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.962277 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4cnj\" (UniqueName: \"kubernetes.io/projected/902dfcac-4523-4a6a-9338-cbe2661ad1ec-kube-api-access-v4cnj\") pod \"node-ca-7wss5\" (UID: \"902dfcac-4523-4a6a-9338-cbe2661ad1ec\") " pod="openshift-image-registry/node-ca-7wss5"
Jul 16 21:50:17.962277 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962210    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-slash\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.962277 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962213    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-multus-socket-dir-parent\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.962277 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962238    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-var-lib-kubelet\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.962277 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962260    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/902dfcac-4523-4a6a-9338-cbe2661ad1ec-host\") pod \"node-ca-7wss5\" (UID: \"902dfcac-4523-4a6a-9338-cbe2661ad1ec\") " pod="openshift-image-registry/node-ca-7wss5"
Jul 16 21:50:17.962655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962283    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwj6g\" (UniqueName: \"kubernetes.io/projected/93848932-9ec1-45e6-aec9-34f0a623b9ef-kube-api-access-fwj6g\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.962655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962307    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65bfdb19-9745-4a5e-842a-05df0b73d7c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.962655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962333    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9vdp\" (UniqueName: \"kubernetes.io/projected/8de28ccc-12c5-4f99-acb9-4e35d577d052-kube-api-access-q9vdp\") pod \"iptables-alerter-rxphl\" (UID: \"8de28ccc-12c5-4f99-acb9-4e35d577d052\") " pod="openshift-network-operator/iptables-alerter-rxphl"
Jul 16 21:50:17.962655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962360    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-hostroot\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.962655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962381    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-run-netns\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.962655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962405    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-etc-openvswitch\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.962655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962429    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93848932-9ec1-45e6-aec9-34f0a623b9ef-ovn-node-metrics-cert\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.962655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962454    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65bfdb19-9745-4a5e-842a-05df0b73d7c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.962655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962471    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-var-lib-cni-bin\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.962655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962505    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:17.962655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962612    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-etc-selinux\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.962655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962633    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-slash\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962666    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-hostroot\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962687    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-var-lib-kubelet\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962505    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-etc-selinux\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962712    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-run-netns\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962731    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-727cj\" (UniqueName: \"kubernetes.io/projected/385e54b7-90e0-441c-8755-be9e7018fca2-kube-api-access-727cj\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962771    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-system-cni-dir\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962797    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-cnibin\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962821    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-multus-conf-dir\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962854    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65bfdb19-9745-4a5e-842a-05df0b73d7c0-os-release\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962877    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-registration-dir\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962879    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65bfdb19-9745-4a5e-842a-05df0b73d7c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962923    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-system-cni-dir\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962942    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-cnibin\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962956    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-systemd-units\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962983    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-node-log\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.962988    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-multus-conf-dir\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963010    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65bfdb19-9745-4a5e-842a-05df0b73d7c0-system-cni-dir\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.963280 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963051    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65bfdb19-9745-4a5e-842a-05df0b73d7c0-os-release\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963038    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8de28ccc-12c5-4f99-acb9-4e35d577d052-iptables-alerter-script\") pod \"iptables-alerter-rxphl\" (UID: \"8de28ccc-12c5-4f99-acb9-4e35d577d052\") " pod="openshift-network-operator/iptables-alerter-rxphl"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963088    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbbaa953-ab0b-4594-a449-06fab72528f5-kubelet-config\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963112    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ac12a88-9208-4759-9c96-3e5de06602a3-konnectivity-ca\") pod \"konnectivity-agent-xn4rj\" (UID: \"4ac12a88-9208-4759-9c96-3e5de06602a3\") " pod="kube-system/konnectivity-agent-xn4rj"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963130    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-multus-cni-dir\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963154    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-var-lib-cni-multus\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963192    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-run-ovn\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963217    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963241    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-cni-netd\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963264    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93848932-9ec1-45e6-aec9-34f0a623b9ef-ovnkube-script-lib\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963286    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65bfdb19-9745-4a5e-842a-05df0b73d7c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963315    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-log-socket\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963333    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-cni-bin\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963355    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93848932-9ec1-45e6-aec9-34f0a623b9ef-env-overrides\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963380    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65bfdb19-9745-4a5e-842a-05df0b73d7c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963452    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-etc-openvswitch\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963521    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ac12a88-9208-4759-9c96-3e5de06602a3-agent-certs\") pod \"konnectivity-agent-xn4rj\" (UID: \"4ac12a88-9208-4759-9c96-3e5de06602a3\") " pod="kube-system/konnectivity-agent-xn4rj"
Jul 16 21:50:17.964216 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963524    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/902dfcac-4523-4a6a-9338-cbe2661ad1ec-host\") pod \"node-ca-7wss5\" (UID: \"902dfcac-4523-4a6a-9338-cbe2661ad1ec\") " pod="openshift-image-registry/node-ca-7wss5"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963555    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-os-release\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963585    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-run-netns\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963608    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/902dfcac-4523-4a6a-9338-cbe2661ad1ec-serviceca\") pod \"node-ca-7wss5\" (UID: \"902dfcac-4523-4a6a-9338-cbe2661ad1ec\") " pod="openshift-image-registry/node-ca-7wss5"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963656    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8de28ccc-12c5-4f99-acb9-4e35d577d052-iptables-alerter-script\") pod \"iptables-alerter-rxphl\" (UID: \"8de28ccc-12c5-4f99-acb9-4e35d577d052\") " pod="openshift-network-operator/iptables-alerter-rxphl"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963633    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963726    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-kubelet\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.963740    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963754    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-run-openvswitch\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963782    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-sys-fs\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.963793    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret podName:cbbaa953-ab0b-4594-a449-06fab72528f5 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:18.46377661 +0000 UTC m=+3.061316836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret") pod "global-pull-secret-syncer-m7rl6" (UID: "cbbaa953-ab0b-4594-a449-06fab72528f5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963824    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-run-k8s-cni-cncf-io\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963840    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-sys-fs\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963853    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-multus-daemon-config\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963880    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93848932-9ec1-45e6-aec9-34f0a623b9ef-ovnkube-config\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963912    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8de28ccc-12c5-4f99-acb9-4e35d577d052-host-slash\") pod \"iptables-alerter-rxphl\" (UID: \"8de28ccc-12c5-4f99-acb9-4e35d577d052\") " pod="openshift-network-operator/iptables-alerter-rxphl"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963918    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-node-log\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.965053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963942    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twhl4\" (UniqueName: \"kubernetes.io/projected/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-kube-api-access-twhl4\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963952    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65bfdb19-9745-4a5e-842a-05df0b73d7c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963959    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65bfdb19-9745-4a5e-842a-05df0b73d7c0-system-cni-dir\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963967    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-run-systemd\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963993    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-kubelet\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963994    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65bfdb19-9745-4a5e-842a-05df0b73d7c0-cnibin\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964029    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-registration-dir\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964047    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65bfdb19-9745-4a5e-842a-05df0b73d7c0-cnibin\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964067    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptz5\" (UniqueName: \"kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5\") pod \"network-check-target-s9mhc\" (UID: \"d5c93b6e-7ae4-4340-86bd-c10c6b804523\") " pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964105    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-socket-dir\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964108    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65bfdb19-9745-4a5e-842a-05df0b73d7c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964170    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964199    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964230    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-cni-netd\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964232    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-device-dir\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964266    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-cni-binary-copy\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964380    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-run-systemd\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.966000 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964383    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbbaa953-ab0b-4594-a449-06fab72528f5-kubelet-config\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964426    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-var-lib-cni-multus\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964443    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-device-dir\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964473    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964511    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-run-ovn\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964513    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-run-k8s-cni-cncf-io\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964544    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-etc-kubernetes\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-var-lib-openvswitch\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.963882    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-systemd-units\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964607    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-run-multus-certs\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964639    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q86pw\" (UniqueName: \"kubernetes.io/projected/65bfdb19-9745-4a5e-842a-05df0b73d7c0-kube-api-access-q86pw\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964666    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbbaa953-ab0b-4594-a449-06fab72528f5-dbus\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964680    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ac12a88-9208-4759-9c96-3e5de06602a3-konnectivity-ca\") pod \"konnectivity-agent-xn4rj\" (UID: \"4ac12a88-9208-4759-9c96-3e5de06602a3\") " pod="kube-system/konnectivity-agent-xn4rj"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964759    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbbaa953-ab0b-4594-a449-06fab72528f5-dbus\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964803    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964853    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-log-socket\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964905    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8de28ccc-12c5-4f99-acb9-4e35d577d052-host-slash\") pod \"iptables-alerter-rxphl\" (UID: \"8de28ccc-12c5-4f99-acb9-4e35d577d052\") " pod="openshift-network-operator/iptables-alerter-rxphl"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964915    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-run-openvswitch\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.966937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964940    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-cni-bin\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964999    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-os-release\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.965139    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-multus-daemon-config\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.965237    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.965248    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93848932-9ec1-45e6-aec9-34f0a623b9ef-ovnkube-config\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.964578    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-multus-cni-dir\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.965282    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-run-netns\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.965328    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-etc-kubernetes\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.965394    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93848932-9ec1-45e6-aec9-34f0a623b9ef-var-lib-openvswitch\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.965451    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-host-run-multus-certs\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.965517    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93848932-9ec1-45e6-aec9-34f0a623b9ef-ovnkube-script-lib\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.965517    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-cni-binary-copy\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.965793    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65bfdb19-9745-4a5e-842a-05df0b73d7c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.965893    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/385e54b7-90e0-441c-8755-be9e7018fca2-socket-dir\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.965878    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93848932-9ec1-45e6-aec9-34f0a623b9ef-env-overrides\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.966221    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/902dfcac-4523-4a6a-9338-cbe2661ad1ec-serviceca\") pod \"node-ca-7wss5\" (UID: \"902dfcac-4523-4a6a-9338-cbe2661ad1ec\") " pod="openshift-image-registry/node-ca-7wss5"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.966750    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93848932-9ec1-45e6-aec9-34f0a623b9ef-ovn-node-metrics-cert\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:17.967740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.967141    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ac12a88-9208-4759-9c96-3e5de06602a3-agent-certs\") pod \"konnectivity-agent-xn4rj\" (UID: \"4ac12a88-9208-4759-9c96-3e5de06602a3\") " pod="kube-system/konnectivity-agent-xn4rj"
Jul 16 21:50:17.992245 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.992151    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:17.992245 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.992198    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:17.992245 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.992215    2570 projected.go:196] Error preparing data for projected volume kube-api-access-vptz5 for pod openshift-network-diagnostics/network-check-target-s9mhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:17.992460 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:17.992284    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5 podName:d5c93b6e-7ae4-4340-86bd-c10c6b804523 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:18.492264212 +0000 UTC m=+3.089804442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vptz5" (UniqueName: "kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5") pod "network-check-target-s9mhc" (UID: "d5c93b6e-7ae4-4340-86bd-c10c6b804523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:17.993909 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.993884    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9vdp\" (UniqueName: \"kubernetes.io/projected/8de28ccc-12c5-4f99-acb9-4e35d577d052-kube-api-access-q9vdp\") pod \"iptables-alerter-rxphl\" (UID: \"8de28ccc-12c5-4f99-acb9-4e35d577d052\") " pod="openshift-network-operator/iptables-alerter-rxphl"
Jul 16 21:50:17.995127 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:17.995086    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-727cj\" (UniqueName: \"kubernetes.io/projected/385e54b7-90e0-441c-8755-be9e7018fca2-kube-api-access-727cj\") pod \"aws-ebs-csi-driver-node-x68v6\" (UID: \"385e54b7-90e0-441c-8755-be9e7018fca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:18.001813 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.001475    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q86pw\" (UniqueName: \"kubernetes.io/projected/65bfdb19-9745-4a5e-842a-05df0b73d7c0-kube-api-access-q86pw\") pod \"multus-additional-cni-plugins-p4gx4\" (UID: \"65bfdb19-9745-4a5e-842a-05df0b73d7c0\") " pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:18.001813 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.001769    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4cnj\" (UniqueName: \"kubernetes.io/projected/902dfcac-4523-4a6a-9338-cbe2661ad1ec-kube-api-access-v4cnj\") pod \"node-ca-7wss5\" (UID: \"902dfcac-4523-4a6a-9338-cbe2661ad1ec\") " pod="openshift-image-registry/node-ca-7wss5"
Jul 16 21:50:18.003765 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.003726    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwj6g\" (UniqueName: \"kubernetes.io/projected/93848932-9ec1-45e6-aec9-34f0a623b9ef-kube-api-access-fwj6g\") pod \"ovnkube-node-r6hvb\" (UID: \"93848932-9ec1-45e6-aec9-34f0a623b9ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:18.004849 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.004341    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twhl4\" (UniqueName: \"kubernetes.io/projected/97b07f25-0e8b-4fe6-8da3-72ca51f28f18-kube-api-access-twhl4\") pod \"multus-njxsf\" (UID: \"97b07f25-0e8b-4fe6-8da3-72ca51f28f18\") " pod="openshift-multus/multus-njxsf"
Jul 16 21:50:18.059238 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.059211    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t54hm"
Jul 16 21:50:18.069803 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.069638    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7wss5"
Jul 16 21:50:18.071542 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:18.071511    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae773a5_b7eb_4346_a16f_186c5f0ec2a8.slice/crio-ea3c001b567bd8dae2f644165e709c0ed2b5020edf62c025a538fe32a25ff9d9 WatchSource:0}: Error finding container ea3c001b567bd8dae2f644165e709c0ed2b5020edf62c025a538fe32a25ff9d9: Status 404 returned error can't find the container with id ea3c001b567bd8dae2f644165e709c0ed2b5020edf62c025a538fe32a25ff9d9
Jul 16 21:50:18.079999 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.079973    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6g6ph"
Jul 16 21:50:18.080812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:18.080789    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902dfcac_4523_4a6a_9338_cbe2661ad1ec.slice/crio-6bdfe0fbb4de09da5873ad6ea55c934c752a8ed3358625bde1be1d8411bacc08 WatchSource:0}: Error finding container 6bdfe0fbb4de09da5873ad6ea55c934c752a8ed3358625bde1be1d8411bacc08: Status 404 returned error can't find the container with id 6bdfe0fbb4de09da5873ad6ea55c934c752a8ed3358625bde1be1d8411bacc08
Jul 16 21:50:18.086065 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.086044    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:18.090414 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:18.090391    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bb9d45_6b80_4455_9c4c_0cab45a12c5f.slice/crio-2e67c833c477443987fb737824188937d8b72f4bb110fb898f3e6577133559b8 WatchSource:0}: Error finding container 2e67c833c477443987fb737824188937d8b72f4bb110fb898f3e6577133559b8: Status 404 returned error can't find the container with id 2e67c833c477443987fb737824188937d8b72f4bb110fb898f3e6577133559b8
Jul 16 21:50:18.094749 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.094727    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p4gx4"
Jul 16 21:50:18.096611 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:18.096584    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93848932_9ec1_45e6_aec9_34f0a623b9ef.slice/crio-6efbde1970904d0b141354510742272738cfa027eb6fd82aa7e5ab2b197c35e6 WatchSource:0}: Error finding container 6efbde1970904d0b141354510742272738cfa027eb6fd82aa7e5ab2b197c35e6: Status 404 returned error can't find the container with id 6efbde1970904d0b141354510742272738cfa027eb6fd82aa7e5ab2b197c35e6
Jul 16 21:50:18.103303 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.103280    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xn4rj"
Jul 16 21:50:18.106273 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:18.106251    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bfdb19_9745_4a5e_842a_05df0b73d7c0.slice/crio-fe6f0a238f4f5ef5541dd29ed9d323523ea7195169aca8a2c4e8ab7793bb511f WatchSource:0}: Error finding container fe6f0a238f4f5ef5541dd29ed9d323523ea7195169aca8a2c4e8ab7793bb511f: Status 404 returned error can't find the container with id fe6f0a238f4f5ef5541dd29ed9d323523ea7195169aca8a2c4e8ab7793bb511f
Jul 16 21:50:18.111998 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.111837    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxphl"
Jul 16 21:50:18.114463 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:18.114439    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac12a88_9208_4759_9c96_3e5de06602a3.slice/crio-3074b542bef7dccd331a33cf09aa11a8964d62386a9bb84fae61cbec40ba3bb1 WatchSource:0}: Error finding container 3074b542bef7dccd331a33cf09aa11a8964d62386a9bb84fae61cbec40ba3bb1: Status 404 returned error can't find the container with id 3074b542bef7dccd331a33cf09aa11a8964d62386a9bb84fae61cbec40ba3bb1
Jul 16 21:50:18.119928 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.119912    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6"
Jul 16 21:50:18.122077 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:18.122055    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de28ccc_12c5_4f99_acb9_4e35d577d052.slice/crio-dad7a814c15f4a83f7e3387909cc5054b4521271efd3c96a4f2177d982ef2015 WatchSource:0}: Error finding container dad7a814c15f4a83f7e3387909cc5054b4521271efd3c96a4f2177d982ef2015: Status 404 returned error can't find the container with id dad7a814c15f4a83f7e3387909cc5054b4521271efd3c96a4f2177d982ef2015
Jul 16 21:50:18.125109 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.125087    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-njxsf"
Jul 16 21:50:18.140077 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:18.139904    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385e54b7_90e0_441c_8755_be9e7018fca2.slice/crio-d6563dc38aae66aecf8fdaef63a10123fb8e5e53e044023bb53332ef20d88bd2 WatchSource:0}: Error finding container d6563dc38aae66aecf8fdaef63a10123fb8e5e53e044023bb53332ef20d88bd2: Status 404 returned error can't find the container with id d6563dc38aae66aecf8fdaef63a10123fb8e5e53e044023bb53332ef20d88bd2
Jul 16 21:50:18.141108 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:18.140995    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b07f25_0e8b_4fe6_8da3_72ca51f28f18.slice/crio-326bcd2a298325f566726951c25db4cde47fba4cfec2be535c7d5afe6562ca05 WatchSource:0}: Error finding container 326bcd2a298325f566726951c25db4cde47fba4cfec2be535c7d5afe6562ca05: Status 404 returned error can't find the container with id 326bcd2a298325f566726951c25db4cde47fba4cfec2be535c7d5afe6562ca05
Jul 16 21:50:18.367778 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.367625    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:18.367941 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:18.367818    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:18.367941 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:18.367899    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs podName:b2365d47-8e55-4776-b882-49eead286f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:50:19.367878256 +0000 UTC m=+3.965418481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs") pod "network-metrics-daemon-c2dgn" (UID: "b2365d47-8e55-4776-b882-49eead286f8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:18.468854 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.468806    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:18.469066 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:18.469035    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:18.469165 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:18.469138    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret podName:cbbaa953-ab0b-4594-a449-06fab72528f5 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:19.469115486 +0000 UTC m=+4.066655698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret") pod "global-pull-secret-syncer-m7rl6" (UID: "cbbaa953-ab0b-4594-a449-06fab72528f5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:18.569870 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.569837    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptz5\" (UniqueName: \"kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5\") pod \"network-check-target-s9mhc\" (UID: \"d5c93b6e-7ae4-4340-86bd-c10c6b804523\") " pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:18.570047 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:18.569997    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:18.570047 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:18.570016    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:18.570047 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:18.570028    2570 projected.go:196] Error preparing data for projected volume kube-api-access-vptz5 for pod openshift-network-diagnostics/network-check-target-s9mhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:18.570220 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:18.570085    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5 podName:d5c93b6e-7ae4-4340-86bd-c10c6b804523 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:19.570066321 +0000 UTC m=+4.167606533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vptz5" (UniqueName: "kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5") pod "network-check-target-s9mhc" (UID: "d5c93b6e-7ae4-4340-86bd-c10c6b804523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:18.850379 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.850250    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:45:18 +0000 UTC" deadline="2027-12-28 06:29:13.657790278 +0000 UTC"
Jul 16 21:50:18.850379 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.850324    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12704h38m54.807472407s"
Jul 16 21:50:18.866723 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.866692    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:18.866860 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:18.866810    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:18.878156 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.878120    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-njxsf" event={"ID":"97b07f25-0e8b-4fe6-8da3-72ca51f28f18","Type":"ContainerStarted","Data":"326bcd2a298325f566726951c25db4cde47fba4cfec2be535c7d5afe6562ca05"}
Jul 16 21:50:18.879603 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.879577    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6" event={"ID":"385e54b7-90e0-441c-8755-be9e7018fca2","Type":"ContainerStarted","Data":"d6563dc38aae66aecf8fdaef63a10123fb8e5e53e044023bb53332ef20d88bd2"}
Jul 16 21:50:18.881099 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.881070    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxphl" event={"ID":"8de28ccc-12c5-4f99-acb9-4e35d577d052","Type":"ContainerStarted","Data":"dad7a814c15f4a83f7e3387909cc5054b4521271efd3c96a4f2177d982ef2015"}
Jul 16 21:50:18.882493 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.882464    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xn4rj" event={"ID":"4ac12a88-9208-4759-9c96-3e5de06602a3","Type":"ContainerStarted","Data":"3074b542bef7dccd331a33cf09aa11a8964d62386a9bb84fae61cbec40ba3bb1"}
Jul 16 21:50:18.883988 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.883959    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4gx4" event={"ID":"65bfdb19-9745-4a5e-842a-05df0b73d7c0","Type":"ContainerStarted","Data":"fe6f0a238f4f5ef5541dd29ed9d323523ea7195169aca8a2c4e8ab7793bb511f"}
Jul 16 21:50:18.885388 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.885352    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb" event={"ID":"93848932-9ec1-45e6-aec9-34f0a623b9ef","Type":"ContainerStarted","Data":"6efbde1970904d0b141354510742272738cfa027eb6fd82aa7e5ab2b197c35e6"}
Jul 16 21:50:18.886680 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.886646    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6g6ph" event={"ID":"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f","Type":"ContainerStarted","Data":"2e67c833c477443987fb737824188937d8b72f4bb110fb898f3e6577133559b8"}
Jul 16 21:50:18.887973 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.887944    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7wss5" event={"ID":"902dfcac-4523-4a6a-9338-cbe2661ad1ec","Type":"ContainerStarted","Data":"6bdfe0fbb4de09da5873ad6ea55c934c752a8ed3358625bde1be1d8411bacc08"}
Jul 16 21:50:18.889198 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:18.889172    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t54hm" event={"ID":"9ae773a5-b7eb-4346-a16f-186c5f0ec2a8","Type":"ContainerStarted","Data":"ea3c001b567bd8dae2f644165e709c0ed2b5020edf62c025a538fe32a25ff9d9"}
Jul 16 21:50:19.376797 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:19.376760    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:19.376986 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:19.376960    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:19.377054 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:19.377026    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs podName:b2365d47-8e55-4776-b882-49eead286f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:50:21.377006863 +0000 UTC m=+5.974547082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs") pod "network-metrics-daemon-c2dgn" (UID: "b2365d47-8e55-4776-b882-49eead286f8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:19.477554 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:19.477515    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:19.477794 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:19.477763    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:19.477876 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:19.477835    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret podName:cbbaa953-ab0b-4594-a449-06fab72528f5 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:21.47781475 +0000 UTC m=+6.075354993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret") pod "global-pull-secret-syncer-m7rl6" (UID: "cbbaa953-ab0b-4594-a449-06fab72528f5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:19.578772 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:19.578132    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptz5\" (UniqueName: \"kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5\") pod \"network-check-target-s9mhc\" (UID: \"d5c93b6e-7ae4-4340-86bd-c10c6b804523\") " pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:19.578772 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:19.578339    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:19.578772 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:19.578360    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:19.578772 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:19.578373    2570 projected.go:196] Error preparing data for projected volume kube-api-access-vptz5 for pod openshift-network-diagnostics/network-check-target-s9mhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:19.578772 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:19.578430    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5 podName:d5c93b6e-7ae4-4340-86bd-c10c6b804523 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:21.578410549 +0000 UTC m=+6.175950765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vptz5" (UniqueName: "kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5") pod "network-check-target-s9mhc" (UID: "d5c93b6e-7ae4-4340-86bd-c10c6b804523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:19.851658 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:19.851388    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:45:18 +0000 UTC" deadline="2027-12-24 03:14:27.658907324 +0000 UTC"
Jul 16 21:50:19.852083 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:19.851660    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12605h24m7.807254216s"
Jul 16 21:50:19.867188 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:19.866704    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:19.867188 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:19.866849    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:19.867374 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:19.867274    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:19.867432 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:19.867376    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:19.901549 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:19.901453    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-85.ec2.internal" event={"ID":"186a905c8a5ab6ff537de73d191d65e4","Type":"ContainerStarted","Data":"15b0573e273c8c444d13bf406e5fefa93baeaaf6218724c5b9be9fc7adf12794"}
Jul 16 21:50:19.920622 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:19.920569    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-85.ec2.internal" podStartSLOduration=1.9205499769999999 podStartE2EDuration="1.920549977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:19.920534098 +0000 UTC m=+4.518074334" watchObservedRunningTime="2026-07-16 21:50:19.920549977 +0000 UTC m=+4.518090203"
Jul 16 21:50:20.867021 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:20.866506    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:20.867021 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:20.866630    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:20.935630 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:20.934999    2570 generic.go:356] "Generic (PLEG): container finished" podID="0d67fa00a9f25b2a9762a6da7434b1b7" containerID="5b60c564e6f0bafe50e016b02ddbc8c5364348e388a9cdb0f4332e6a060b8588" exitCode=0
Jul 16 21:50:20.935630 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:20.935534    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal" event={"ID":"0d67fa00a9f25b2a9762a6da7434b1b7","Type":"ContainerDied","Data":"5b60c564e6f0bafe50e016b02ddbc8c5364348e388a9cdb0f4332e6a060b8588"}
Jul 16 21:50:21.396709 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:21.396665    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:21.396906 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:21.396837    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:21.396980 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:21.396922    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs podName:b2365d47-8e55-4776-b882-49eead286f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:50:25.39689979 +0000 UTC m=+9.994440009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs") pod "network-metrics-daemon-c2dgn" (UID: "b2365d47-8e55-4776-b882-49eead286f8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:21.498041 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:21.497416    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:21.498041 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:21.497579    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:21.498041 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:21.497642    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret podName:cbbaa953-ab0b-4594-a449-06fab72528f5 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:25.497622345 +0000 UTC m=+10.095162563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret") pod "global-pull-secret-syncer-m7rl6" (UID: "cbbaa953-ab0b-4594-a449-06fab72528f5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:21.598906 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:21.598216    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptz5\" (UniqueName: \"kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5\") pod \"network-check-target-s9mhc\" (UID: \"d5c93b6e-7ae4-4340-86bd-c10c6b804523\") " pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:21.598906 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:21.598442    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:21.598906 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:21.598476    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:21.598906 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:21.598489    2570 projected.go:196] Error preparing data for projected volume kube-api-access-vptz5 for pod openshift-network-diagnostics/network-check-target-s9mhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:21.598906 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:21.598546    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5 podName:d5c93b6e-7ae4-4340-86bd-c10c6b804523 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:25.598528051 +0000 UTC m=+10.196068282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vptz5" (UniqueName: "kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5") pod "network-check-target-s9mhc" (UID: "d5c93b6e-7ae4-4340-86bd-c10c6b804523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:21.872107 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:21.872009    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:21.872558 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:21.872152    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:21.872558 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:21.872555    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:21.872666 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:21.872642    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:22.866626 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:22.866587    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:22.866825 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:22.866727    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:23.868823 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:23.868792    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:23.869206 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:23.868930    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:23.869390 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:23.869369    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:23.869521 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:23.869488    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:24.866394 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:24.866361    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:24.866599 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:24.866489    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:25.431866 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:25.431830    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:25.432381 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:25.432078    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:25.432381 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:25.432165    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs podName:b2365d47-8e55-4776-b882-49eead286f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:50:33.432142344 +0000 UTC m=+18.029682566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs") pod "network-metrics-daemon-c2dgn" (UID: "b2365d47-8e55-4776-b882-49eead286f8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:25.533812 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:25.533229    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:25.533812 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:25.533422    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:25.533812 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:25.533481    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret podName:cbbaa953-ab0b-4594-a449-06fab72528f5 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:33.533462671 +0000 UTC m=+18.131002890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret") pod "global-pull-secret-syncer-m7rl6" (UID: "cbbaa953-ab0b-4594-a449-06fab72528f5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:25.633976 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:25.633841    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptz5\" (UniqueName: \"kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5\") pod \"network-check-target-s9mhc\" (UID: \"d5c93b6e-7ae4-4340-86bd-c10c6b804523\") " pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:25.634178 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:25.634025    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:25.634178 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:25.634068    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:25.634178 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:25.634086    2570 projected.go:196] Error preparing data for projected volume kube-api-access-vptz5 for pod openshift-network-diagnostics/network-check-target-s9mhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:25.634178 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:25.634173    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5 podName:d5c93b6e-7ae4-4340-86bd-c10c6b804523 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:33.634150526 +0000 UTC m=+18.231690762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vptz5" (UniqueName: "kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5") pod "network-check-target-s9mhc" (UID: "d5c93b6e-7ae4-4340-86bd-c10c6b804523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:25.867452 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:25.867287    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:25.867452 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:25.867382    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:25.867452 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:25.867404    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:25.867793 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:25.867509    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:26.867101 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:26.867054    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:26.867548 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:26.867197    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:27.867163 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:27.867124    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:27.867591 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:27.867272    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:27.867662 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:27.867124    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:27.867777 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:27.867733    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:28.866394 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:28.866370    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:28.866513 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:28.866458    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:29.867139 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:29.867107    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:29.867546 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:29.867213    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:29.867546 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:29.867284    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:29.867546 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:29.867443    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:30.867103 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:30.867056    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:30.867279 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:30.867180    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:31.866974 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:31.866945    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:31.867178 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:31.867058    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:31.867178 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:31.867145    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:31.867290 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:31.867250    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:32.866752 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:32.866725    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:32.866962 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:32.866835    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:33.495498 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:33.495463    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:33.495932 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:33.495604    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:33.495932 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:33.495677    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs podName:b2365d47-8e55-4776-b882-49eead286f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:50:49.495658146 +0000 UTC m=+34.093198355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs") pod "network-metrics-daemon-c2dgn" (UID: "b2365d47-8e55-4776-b882-49eead286f8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:33.596735 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:33.596700    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:33.596874 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:33.596820    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:33.596938 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:33.596891    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret podName:cbbaa953-ab0b-4594-a449-06fab72528f5 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:49.596872191 +0000 UTC m=+34.194412409 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret") pod "global-pull-secret-syncer-m7rl6" (UID: "cbbaa953-ab0b-4594-a449-06fab72528f5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:33.697847 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:33.697812    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptz5\" (UniqueName: \"kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5\") pod \"network-check-target-s9mhc\" (UID: \"d5c93b6e-7ae4-4340-86bd-c10c6b804523\") " pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:33.697984 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:33.697962    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:33.697984 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:33.697978    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:33.698051 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:33.697988    2570 projected.go:196] Error preparing data for projected volume kube-api-access-vptz5 for pod openshift-network-diagnostics/network-check-target-s9mhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:33.698051 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:33.698034    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5 podName:d5c93b6e-7ae4-4340-86bd-c10c6b804523 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:49.698020998 +0000 UTC m=+34.295561214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vptz5" (UniqueName: "kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5") pod "network-check-target-s9mhc" (UID: "d5c93b6e-7ae4-4340-86bd-c10c6b804523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:33.866811 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:33.866733    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:33.866965 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:33.866741    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:33.866965 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:33.866870    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:33.867053 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:33.866966    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:34.866756 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:34.866713    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:34.867137 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:34.866844    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:35.867511 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:35.867481    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:35.867954 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:35.867589    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:35.867954 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:35.867680    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:35.867954 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:35.867778    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:36.866096 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:36.866062    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:36.866267 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:36.866207    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:37.866705 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:37.866675    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:37.867122 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:37.866798    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:37.867122 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:37.866835    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:37.867122 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:37.866888    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:38.867011 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.866777    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:38.867661 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:38.867118    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:38.974178 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.974144    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal" event={"ID":"0d67fa00a9f25b2a9762a6da7434b1b7","Type":"ContainerStarted","Data":"da8aea20267a4753a5ee1366f1b6209023fcdee500d94cf4dee1509a0c1d0375"}
Jul 16 21:50:38.975734 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.975711    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-njxsf" event={"ID":"97b07f25-0e8b-4fe6-8da3-72ca51f28f18","Type":"ContainerStarted","Data":"d86823fe912c9a218b5df3af7bbfe55f2a899e4bca1009a7acd64295bd9a2fce"}
Jul 16 21:50:38.977314 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.977294    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6" event={"ID":"385e54b7-90e0-441c-8755-be9e7018fca2","Type":"ContainerStarted","Data":"403c35f35049a0e39aab8f1f61ecfd379f8a19512894ca9c2410e88d92ac5de0"}
Jul 16 21:50:38.978857 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.978837    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xn4rj" event={"ID":"4ac12a88-9208-4759-9c96-3e5de06602a3","Type":"ContainerStarted","Data":"38fc470715102fc4d2ad7e3bad2371bd04dd3b21960cb9af63e0f03b493a438f"}
Jul 16 21:50:38.980722 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.980697    2570 generic.go:356] "Generic (PLEG): container finished" podID="65bfdb19-9745-4a5e-842a-05df0b73d7c0" containerID="61aa02b91a277fc99919bcc499fcd1c9987fd5db046eed46acfd5eb84ee36846" exitCode=0
Jul 16 21:50:38.980824 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.980725    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4gx4" event={"ID":"65bfdb19-9745-4a5e-842a-05df0b73d7c0","Type":"ContainerDied","Data":"61aa02b91a277fc99919bcc499fcd1c9987fd5db046eed46acfd5eb84ee36846"}
Jul 16 21:50:38.984259 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.984243    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 21:50:38.984562 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.984544    2570 generic.go:356] "Generic (PLEG): container finished" podID="93848932-9ec1-45e6-aec9-34f0a623b9ef" containerID="a6c4825ef8d8bce69c261d1edd4a2cedd86e6c7ad4bf95b58639c9c2dfcd0bdf" exitCode=1
Jul 16 21:50:38.984630 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.984598    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb" event={"ID":"93848932-9ec1-45e6-aec9-34f0a623b9ef","Type":"ContainerStarted","Data":"c63edfa6a6edd8e800262db8d6429bb4eedbb10047468c1b38131d5e56a25f98"}
Jul 16 21:50:38.984671 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.984649    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb" event={"ID":"93848932-9ec1-45e6-aec9-34f0a623b9ef","Type":"ContainerStarted","Data":"f09fb5edd37e60da7ffa30d96057cfae305a444a3ea46111dfece00e48dcd221"}
Jul 16 21:50:38.984718 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.984688    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb" event={"ID":"93848932-9ec1-45e6-aec9-34f0a623b9ef","Type":"ContainerStarted","Data":"1434c7bc9585c0a37fe450e1ea643854ce236e874b3f43a4a108b2583af89196"}
Jul 16 21:50:38.984757 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.984730    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb" event={"ID":"93848932-9ec1-45e6-aec9-34f0a623b9ef","Type":"ContainerStarted","Data":"790ee6171eafc0e6d559eea818eb335623d47752387fc3b1d4e4e1bf1421400d"}
Jul 16 21:50:38.984797 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.984757    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb" event={"ID":"93848932-9ec1-45e6-aec9-34f0a623b9ef","Type":"ContainerDied","Data":"a6c4825ef8d8bce69c261d1edd4a2cedd86e6c7ad4bf95b58639c9c2dfcd0bdf"}
Jul 16 21:50:38.984839 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.984796    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb" event={"ID":"93848932-9ec1-45e6-aec9-34f0a623b9ef","Type":"ContainerStarted","Data":"c190a3c28ed9ae38eaa39bbd062b4d5df3593f2609264b06cc3f3851096fc53c"}
Jul 16 21:50:38.986252 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.986230    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6g6ph" event={"ID":"d9bb9d45-6b80-4455-9c4c-0cab45a12c5f","Type":"ContainerStarted","Data":"ff307562874553525c7faef9b08706562bcd7f2e387646ae0d1af56e70c10f9f"}
Jul 16 21:50:38.987915 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.987891    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7wss5" event={"ID":"902dfcac-4523-4a6a-9338-cbe2661ad1ec","Type":"ContainerStarted","Data":"b8208effd60e68b5d0f2610c188cbd0500808110cf0ac7cbf5b840ea454aa450"}
Jul 16 21:50:38.989426 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.989396    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t54hm" event={"ID":"9ae773a5-b7eb-4346-a16f-186c5f0ec2a8","Type":"ContainerStarted","Data":"671c0a56351e855b84e9e9f5c3e2aa59b9204a0f17e0cc1491ff416e7fcebfd7"}
Jul 16 21:50:38.997910 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:38.994806    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-85.ec2.internal" podStartSLOduration=20.994783024 podStartE2EDuration="20.994783024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:38.993232741 +0000 UTC m=+23.590772963" watchObservedRunningTime="2026-07-16 21:50:38.994783024 +0000 UTC m=+23.592323255"
Jul 16 21:50:39.011310 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.011275    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t54hm" podStartSLOduration=3.004610749 podStartE2EDuration="23.011265141s" totalImagesPullingTime="20.006654392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:39.010909833 +0000 UTC m=+23.608450064" watchObservedRunningTime="2026-07-16 21:50:39.011265141 +0000 UTC m=+23.608805372"
Jul 16 21:50:39.063999 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.063958    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-njxsf" podStartSLOduration=3.108600328 podStartE2EDuration="23.063944915s" totalImagesPullingTime="19.955344587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:39.063658711 +0000 UTC m=+23.661198941" watchObservedRunningTime="2026-07-16 21:50:39.063944915 +0000 UTC m=+23.661485165"
Jul 16 21:50:39.110323 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.110273    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6g6ph" podStartSLOduration=3.122372703 podStartE2EDuration="23.110258001s" totalImagesPullingTime="19.987885298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:39.086676849 +0000 UTC m=+23.684217080" watchObservedRunningTime="2026-07-16 21:50:39.110258001 +0000 UTC m=+23.707798233"
Jul 16 21:50:39.110454 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.110399    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7wss5" podStartSLOduration=3.11630303 podStartE2EDuration="23.110390182s" totalImagesPullingTime="19.994087152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:39.110322101 +0000 UTC m=+23.707862334" watchObservedRunningTime="2026-07-16 21:50:39.110390182 +0000 UTC m=+23.707930416"
Jul 16 21:50:39.133053 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.132618    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xn4rj" podStartSLOduration=3.171265148 podStartE2EDuration="23.132603085s" totalImagesPullingTime="19.961337937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:39.132527905 +0000 UTC m=+23.730068138" watchObservedRunningTime="2026-07-16 21:50:39.132603085 +0000 UTC m=+23.730143317"
Jul 16 21:50:39.503597 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.503575    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 21:50:39.821394 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.821278    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:50:39.50359357Z","UUID":"62dfa60d-2cb5-4e76-8662-8e1b5d8c8afb","Handler":null,"Name":"","Endpoint":""}
Jul 16 21:50:39.823119 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.823096    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 21:50:39.823252 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.823142    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 21:50:39.867292 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.867266    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:39.867694 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:39.867412    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:39.867694 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.867474    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:39.867694 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:39.867577    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:39.993744 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.993710    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6" event={"ID":"385e54b7-90e0-441c-8755-be9e7018fca2","Type":"ContainerStarted","Data":"4ecc636b63e495f303b0b58e44be41a29c75f25a59b873bd6fd642b6736a55da"}
Jul 16 21:50:39.995481 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:39.995454    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxphl" event={"ID":"8de28ccc-12c5-4f99-acb9-4e35d577d052","Type":"ContainerStarted","Data":"327dc9fd6c68e50340660f8da0688c1fe442b3edcad0b6ecd92152b219c9fae0"}
Jul 16 21:50:40.015720 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:40.015633    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rxphl" podStartSLOduration=4.061351107 podStartE2EDuration="24.015613865s" totalImagesPullingTime="19.954262758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:40.015211219 +0000 UTC m=+24.612751448" watchObservedRunningTime="2026-07-16 21:50:40.015613865 +0000 UTC m=+24.613154100"
Jul 16 21:50:40.866792 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:40.866720    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:40.866933 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:40.866819    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:40.999430 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:40.999404    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6" event={"ID":"385e54b7-90e0-441c-8755-be9e7018fca2","Type":"ContainerStarted","Data":"ee458e7a3fc5222e04d025a489421a216b5046cd2b2acb9892879cc5266a967c"}
Jul 16 21:50:41.047974 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:41.047934    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x68v6" podStartSLOduration=2.9107529960000003 podStartE2EDuration="25.047915569s" totalImagesPullingTime="22.137162573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:41.047820242 +0000 UTC m=+25.645360476" watchObservedRunningTime="2026-07-16 21:50:41.047915569 +0000 UTC m=+25.645455831"
Jul 16 21:50:41.867346 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:41.867159    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:41.867545 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:41.867207    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:41.867545 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:41.867433    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:41.867659 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:41.867538    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:42.004630 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:42.004603    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 21:50:42.005095 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:42.005061    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb" event={"ID":"93848932-9ec1-45e6-aec9-34f0a623b9ef","Type":"ContainerStarted","Data":"1c7482f3d3ed632f33ec45fd917a963ab94649162f1f75fa151103f1376b7100"}
Jul 16 21:50:42.866981 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:42.866951    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:42.867154 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:42.867058    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:43.823119 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:43.822707    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xn4rj"
Jul 16 21:50:43.823984 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:43.823424    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xn4rj"
Jul 16 21:50:43.867037 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:43.867012    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:43.867161 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:43.867131    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:43.867213 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:43.867187    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:43.867300 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:43.867278    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:44.010332 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:44.010304    2570 generic.go:356] "Generic (PLEG): container finished" podID="65bfdb19-9745-4a5e-842a-05df0b73d7c0" containerID="47e4ad0e507b26799836d6a35bb011e69d9445dd68367c19c3ba8914d46bc205" exitCode=0
Jul 16 21:50:44.010453 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:44.010382    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4gx4" event={"ID":"65bfdb19-9745-4a5e-842a-05df0b73d7c0","Type":"ContainerDied","Data":"47e4ad0e507b26799836d6a35bb011e69d9445dd68367c19c3ba8914d46bc205"}
Jul 16 21:50:44.015937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:44.015919    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 21:50:44.016288 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:44.016264    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb" event={"ID":"93848932-9ec1-45e6-aec9-34f0a623b9ef","Type":"ContainerStarted","Data":"e7843c8ec4864a15ab20e39d57ca81739e7ff79252b434fadf526fcec67b80d1"}
Jul 16 21:50:44.016745 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:44.016727    2570 scope.go:117] "RemoveContainer" containerID="a6c4825ef8d8bce69c261d1edd4a2cedd86e6c7ad4bf95b58639c9c2dfcd0bdf"
Jul 16 21:50:44.866258 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:44.866229    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:44.866905 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:44.866325    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:45.023824 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.023802    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 21:50:45.024232 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.024203    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb" event={"ID":"93848932-9ec1-45e6-aec9-34f0a623b9ef","Type":"ContainerStarted","Data":"77d358c5a1dd314e85c8c9f609ee35740986e9483141a3bbe052b1a4b758bf06"}
Jul 16 21:50:45.024657 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.024504    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:45.024657 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.024575    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:45.024657 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.024621    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:45.040315 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.040293    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:45.041199 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.041183    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:50:45.067464 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.067426    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb" podStartSLOduration=8.8724533 podStartE2EDuration="29.067413738s" totalImagesPullingTime="20.194960438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:45.062907098 +0000 UTC m=+29.660447329" watchObservedRunningTime="2026-07-16 21:50:45.067413738 +0000 UTC m=+29.664953970"
Jul 16 21:50:45.258676 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.258650    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m7rl6"]
Jul 16 21:50:45.258797 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.258771    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:45.258870 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:45.258851    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:45.260711 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.260686    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c2dgn"]
Jul 16 21:50:45.260830 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.260808    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:45.260962 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:45.260937    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:45.261320 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.261302    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s9mhc"]
Jul 16 21:50:45.261392 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:45.261381    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:45.261508 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:45.261483    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:46.866419 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:46.866240    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:46.866887 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:46.866287    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:46.866887 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:46.866502    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:46.866887 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:46.866299    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:46.866887 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:46.866552    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:46.866887 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:46.866609    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:47.030689 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:47.030665    2570 generic.go:356] "Generic (PLEG): container finished" podID="65bfdb19-9745-4a5e-842a-05df0b73d7c0" containerID="cd1f583056078ad160560e61af957ea6c43341a37bb0b1f63a81f8e2884f6bef" exitCode=0
Jul 16 21:50:47.030835 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:47.030720    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4gx4" event={"ID":"65bfdb19-9745-4a5e-842a-05df0b73d7c0","Type":"ContainerDied","Data":"cd1f583056078ad160560e61af957ea6c43341a37bb0b1f63a81f8e2884f6bef"}
Jul 16 21:50:47.423995 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:47.423969    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xn4rj"
Jul 16 21:50:47.424149 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:47.424126    2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 16 21:50:47.424602 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:47.424585    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xn4rj"
Jul 16 21:50:48.866747 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:48.866719    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:48.867316 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:48.866773    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:48.867316 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:48.866783    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:48.867316 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:48.866845    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2dgn" podUID="b2365d47-8e55-4776-b882-49eead286f8c"
Jul 16 21:50:48.867316 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:48.866903    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7rl6" podUID="cbbaa953-ab0b-4594-a449-06fab72528f5"
Jul 16 21:50:48.867316 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:48.866967    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9mhc" podUID="d5c93b6e-7ae4-4340-86bd-c10c6b804523"
Jul 16 21:50:49.036976 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:49.036950    2570 generic.go:356] "Generic (PLEG): container finished" podID="65bfdb19-9745-4a5e-842a-05df0b73d7c0" containerID="6bdde1b5acad8488aa6b5a6d2f850d7b0eb5152778c9d7bc9085cf18c0346d6a" exitCode=0
Jul 16 21:50:49.037133 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:49.036999    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4gx4" event={"ID":"65bfdb19-9745-4a5e-842a-05df0b73d7c0","Type":"ContainerDied","Data":"6bdde1b5acad8488aa6b5a6d2f850d7b0eb5152778c9d7bc9085cf18c0346d6a"}
Jul 16 21:50:49.509041 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:49.509006    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:49.509210 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:49.509175    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:49.509295 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:49.509248    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs podName:b2365d47-8e55-4776-b882-49eead286f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:51:21.509229304 +0000 UTC m=+66.106769527 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs") pod "network-metrics-daemon-c2dgn" (UID: "b2365d47-8e55-4776-b882-49eead286f8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:49.609585 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:49.609559    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:49.609699 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:49.609685    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:49.609746 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:49.609733    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret podName:cbbaa953-ab0b-4594-a449-06fab72528f5 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:21.609719447 +0000 UTC m=+66.207259660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret") pod "global-pull-secret-syncer-m7rl6" (UID: "cbbaa953-ab0b-4594-a449-06fab72528f5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:49.710615 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:49.710589    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptz5\" (UniqueName: \"kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5\") pod \"network-check-target-s9mhc\" (UID: \"d5c93b6e-7ae4-4340-86bd-c10c6b804523\") " pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:49.710796 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:49.710725    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:49.710796 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:49.710743    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:49.710796 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:49.710759    2570 projected.go:196] Error preparing data for projected volume kube-api-access-vptz5 for pod openshift-network-diagnostics/network-check-target-s9mhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:49.710950 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:49.710807    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5 podName:d5c93b6e-7ae4-4340-86bd-c10c6b804523 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:21.710794032 +0000 UTC m=+66.308334242 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vptz5" (UniqueName: "kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5") pod "network-check-target-s9mhc" (UID: "d5c93b6e-7ae4-4340-86bd-c10c6b804523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:50.727198 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.727171    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-85.ec2.internal" event="NodeReady"
Jul 16 21:50:50.727603 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.727309    2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 21:50:50.779032 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.779003    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6b842"]
Jul 16 21:50:50.781022 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.781001    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zq96d"]
Jul 16 21:50:50.781178 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.781159    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:50.782985 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.782962    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zq96d"
Jul 16 21:50:50.783751 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.783731    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 21:50:50.786525 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.786492    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q4vmw\""
Jul 16 21:50:50.786670 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.786642    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 21:50:50.787291 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.787058    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-khfb2\""
Jul 16 21:50:50.787291 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.787058    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 21:50:50.787291 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.787206    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 21:50:50.787465 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.787322    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 21:50:50.790976 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.790955    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6b842"]
Jul 16 21:50:50.795128 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.795108    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zq96d"]
Jul 16 21:50:50.866298 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.866240    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:50:50.866393 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.866355    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:50:50.866457 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.866391    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:50:50.868802 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.868783    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:50:50.868990 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.868952    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:50:50.869241 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.869173    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:50:50.869493 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.869471    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wslpm\""
Jul 16 21:50:50.869493 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.869486    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fdwm\""
Jul 16 21:50:50.869638 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.869571    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:50:50.920682 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.920651    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:50.920792 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.920694    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a6451d-57f1-412b-9f81-e2903dfdff69-config-volume\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:50.920792 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.920731    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwjsb\" (UniqueName: \"kubernetes.io/projected/ea5a4ca6-5adf-47f1-a1e4-4c59f554714d-kube-api-access-bwjsb\") pod \"ingress-canary-zq96d\" (UID: \"ea5a4ca6-5adf-47f1-a1e4-4c59f554714d\") " pod="openshift-ingress-canary/ingress-canary-zq96d"
Jul 16 21:50:50.920895 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.920786    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84rqn\" (UniqueName: \"kubernetes.io/projected/f8a6451d-57f1-412b-9f81-e2903dfdff69-kube-api-access-84rqn\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:50.920895 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.920857    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a6451d-57f1-412b-9f81-e2903dfdff69-tmp-dir\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:50.920895 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:50.920884    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea5a4ca6-5adf-47f1-a1e4-4c59f554714d-cert\") pod \"ingress-canary-zq96d\" (UID: \"ea5a4ca6-5adf-47f1-a1e4-4c59f554714d\") " pod="openshift-ingress-canary/ingress-canary-zq96d"
Jul 16 21:50:51.021748 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.021724    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea5a4ca6-5adf-47f1-a1e4-4c59f554714d-cert\") pod \"ingress-canary-zq96d\" (UID: \"ea5a4ca6-5adf-47f1-a1e4-4c59f554714d\") " pod="openshift-ingress-canary/ingress-canary-zq96d"
Jul 16 21:50:51.021883 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.021776    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:51.021883 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:51.021857    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:51.022001 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:51.021912    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls podName:f8a6451d-57f1-412b-9f81-e2903dfdff69 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:51.521894351 +0000 UTC m=+36.119434580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls") pod "dns-default-6b842" (UID: "f8a6451d-57f1-412b-9f81-e2903dfdff69") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:51.022001 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.021929    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a6451d-57f1-412b-9f81-e2903dfdff69-config-volume\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:51.022001 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.021968    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjsb\" (UniqueName: \"kubernetes.io/projected/ea5a4ca6-5adf-47f1-a1e4-4c59f554714d-kube-api-access-bwjsb\") pod \"ingress-canary-zq96d\" (UID: \"ea5a4ca6-5adf-47f1-a1e4-4c59f554714d\") " pod="openshift-ingress-canary/ingress-canary-zq96d"
Jul 16 21:50:51.022001 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.021992    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rqn\" (UniqueName: \"kubernetes.io/projected/f8a6451d-57f1-412b-9f81-e2903dfdff69-kube-api-access-84rqn\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:51.022273 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.022043    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a6451d-57f1-412b-9f81-e2903dfdff69-tmp-dir\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:51.022431 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.022398    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a6451d-57f1-412b-9f81-e2903dfdff69-tmp-dir\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:51.022816 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.022795    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a6451d-57f1-412b-9f81-e2903dfdff69-config-volume\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:51.026725 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.026705    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea5a4ca6-5adf-47f1-a1e4-4c59f554714d-cert\") pod \"ingress-canary-zq96d\" (UID: \"ea5a4ca6-5adf-47f1-a1e4-4c59f554714d\") " pod="openshift-ingress-canary/ingress-canary-zq96d"
Jul 16 21:50:51.030667 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.030646    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rqn\" (UniqueName: \"kubernetes.io/projected/f8a6451d-57f1-412b-9f81-e2903dfdff69-kube-api-access-84rqn\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:51.031013 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.030995    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjsb\" (UniqueName: \"kubernetes.io/projected/ea5a4ca6-5adf-47f1-a1e4-4c59f554714d-kube-api-access-bwjsb\") pod \"ingress-canary-zq96d\" (UID: \"ea5a4ca6-5adf-47f1-a1e4-4c59f554714d\") " pod="openshift-ingress-canary/ingress-canary-zq96d"
Jul 16 21:50:51.100527 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.100498    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zq96d"
Jul 16 21:50:51.246451 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.246420    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zq96d"]
Jul 16 21:50:51.250010 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:51.249976    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5a4ca6_5adf_47f1_a1e4_4c59f554714d.slice/crio-a9ed908d478d831a10e09ea07e4be39b2253fefee8b778a42ef869a853957d71 WatchSource:0}: Error finding container a9ed908d478d831a10e09ea07e4be39b2253fefee8b778a42ef869a853957d71: Status 404 returned error can't find the container with id a9ed908d478d831a10e09ea07e4be39b2253fefee8b778a42ef869a853957d71
Jul 16 21:50:51.525930 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.525899    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:51.526064 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:51.526016    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:51.526140 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:51.526068    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls podName:f8a6451d-57f1-412b-9f81-e2903dfdff69 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:52.526054253 +0000 UTC m=+37.123594463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls") pod "dns-default-6b842" (UID: "f8a6451d-57f1-412b-9f81-e2903dfdff69") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:51.692937 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.692903    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r8jdg"]
Jul 16 21:50:51.698379 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.698358    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.703917 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.703881    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 21:50:51.704030 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.703959    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s89fr\""
Jul 16 21:50:51.704030 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.703981    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 21:50:51.704792 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.704774    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 21:50:51.704792 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.704784    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 21:50:51.712832 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.712810    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r8jdg"]
Jul 16 21:50:51.827168 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.827101    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.827168 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.827142    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrfk\" (UniqueName: \"kubernetes.io/projected/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-api-access-7lrfk\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.827677 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.827229    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.827677 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.827280    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-crio-socket\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.827677 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.827303    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-data-volume\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.927711 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.927681    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrfk\" (UniqueName: \"kubernetes.io/projected/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-api-access-7lrfk\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.927871 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.927752    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.927871 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.927781    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-crio-socket\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.927871 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.927800    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-data-volume\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.927871 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.927840    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.928003 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:51.927923    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:50:51.928003 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:51.927937    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:51.928003 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:51.927980    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm podName:8559cfa0-b3f8-4719-9e05-cef5ed2a5aec nodeName:}" failed. No retries permitted until 2026-07-16 21:50:52.427966143 +0000 UTC m=+37.025506351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r8jdg" (UID: "8559cfa0-b3f8-4719-9e05-cef5ed2a5aec") : configmap "kube-rbac-proxy" not found
Jul 16 21:50:51.928003 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:51.927994    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls podName:8559cfa0-b3f8-4719-9e05-cef5ed2a5aec nodeName:}" failed. No retries permitted until 2026-07-16 21:50:52.427987626 +0000 UTC m=+37.025527835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r8jdg" (UID: "8559cfa0-b3f8-4719-9e05-cef5ed2a5aec") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:51.928162 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.928062    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-crio-socket\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.928309 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.928289    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-data-volume\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:51.940948 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:51.940920    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrfk\" (UniqueName: \"kubernetes.io/projected/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-api-access-7lrfk\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:52.045222 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:52.045184    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zq96d" event={"ID":"ea5a4ca6-5adf-47f1-a1e4-4c59f554714d","Type":"ContainerStarted","Data":"a9ed908d478d831a10e09ea07e4be39b2253fefee8b778a42ef869a853957d71"}
Jul 16 21:50:52.431351 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:52.431306    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:52.431517 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:52.431398    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:52.431517 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:52.431440    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:52.431517 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:52.431506    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls podName:8559cfa0-b3f8-4719-9e05-cef5ed2a5aec nodeName:}" failed. No retries permitted until 2026-07-16 21:50:53.431487669 +0000 UTC m=+38.029027886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r8jdg" (UID: "8559cfa0-b3f8-4719-9e05-cef5ed2a5aec") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:52.431688 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:52.431544    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:50:52.431688 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:52.431613    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm podName:8559cfa0-b3f8-4719-9e05-cef5ed2a5aec nodeName:}" failed. No retries permitted until 2026-07-16 21:50:53.431597697 +0000 UTC m=+38.029137930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r8jdg" (UID: "8559cfa0-b3f8-4719-9e05-cef5ed2a5aec") : configmap "kube-rbac-proxy" not found
Jul 16 21:50:52.532150 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:52.532113    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:52.532330 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:52.532291    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:52.532397 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:52.532375    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls podName:f8a6451d-57f1-412b-9f81-e2903dfdff69 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:54.532354012 +0000 UTC m=+39.129894229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls") pod "dns-default-6b842" (UID: "f8a6451d-57f1-412b-9f81-e2903dfdff69") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:53.440355 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:53.440329    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:53.440740 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:53.440400    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:53.440740 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:53.440455    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:53.440740 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:53.440488    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:50:53.440740 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:53.440522    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls podName:8559cfa0-b3f8-4719-9e05-cef5ed2a5aec nodeName:}" failed. No retries permitted until 2026-07-16 21:50:55.440503267 +0000 UTC m=+40.038043493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r8jdg" (UID: "8559cfa0-b3f8-4719-9e05-cef5ed2a5aec") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:53.440740 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:53.440538    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm podName:8559cfa0-b3f8-4719-9e05-cef5ed2a5aec nodeName:}" failed. No retries permitted until 2026-07-16 21:50:55.440529981 +0000 UTC m=+40.038070194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r8jdg" (UID: "8559cfa0-b3f8-4719-9e05-cef5ed2a5aec") : configmap "kube-rbac-proxy" not found
Jul 16 21:50:53.737361 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:53.737127    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t54hm_9ae773a5-b7eb-4346-a16f-186c5f0ec2a8/dns-node-resolver/0.log"
Jul 16 21:50:54.053542 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:54.053459    2570 generic.go:356] "Generic (PLEG): container finished" podID="65bfdb19-9745-4a5e-842a-05df0b73d7c0" containerID="da154c98b506189f72b209dacdadc2c7a7393efd7323f6f8097698ec47ddcc8a" exitCode=0
Jul 16 21:50:54.053542 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:54.053512    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4gx4" event={"ID":"65bfdb19-9745-4a5e-842a-05df0b73d7c0","Type":"ContainerDied","Data":"da154c98b506189f72b209dacdadc2c7a7393efd7323f6f8097698ec47ddcc8a"}
Jul 16 21:50:54.537672 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:54.537642    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7wss5_902dfcac-4523-4a6a-9338-cbe2661ad1ec/node-ca/0.log"
Jul 16 21:50:54.549951 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:54.549918    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:54.550154 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:54.550130    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:54.550275 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:54.550204    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls podName:f8a6451d-57f1-412b-9f81-e2903dfdff69 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:58.550182773 +0000 UTC m=+43.147723028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls") pod "dns-default-6b842" (UID: "f8a6451d-57f1-412b-9f81-e2903dfdff69") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:55.058491 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.058464    2570 generic.go:356] "Generic (PLEG): container finished" podID="65bfdb19-9745-4a5e-842a-05df0b73d7c0" containerID="20f7482cc788092aea76a25dca0d4727678872a6a213db0466c622289e412ab7" exitCode=0
Jul 16 21:50:55.058640 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.058509    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4gx4" event={"ID":"65bfdb19-9745-4a5e-842a-05df0b73d7c0","Type":"ContainerDied","Data":"20f7482cc788092aea76a25dca0d4727678872a6a213db0466c622289e412ab7"}
Jul 16 21:50:55.346324 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.346296    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-g27zm"]
Jul 16 21:50:55.356691 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.356673    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-g27zm"
Jul 16 21:50:55.358983 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.358959    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-g27zm"]
Jul 16 21:50:55.359062 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.359044    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 16 21:50:55.359921 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.359899    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 16 21:50:55.360013 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.359926    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-w4hfs\""
Jul 16 21:50:55.360013 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.359908    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 16 21:50:55.360383 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.360367    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 16 21:50:55.456237 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.456204    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:55.456435 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.456257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7b7695cb-b318-4e8c-a5bf-5dd7faa13223-signing-key\") pod \"service-ca-58598695f7-g27zm\" (UID: \"7b7695cb-b318-4e8c-a5bf-5dd7faa13223\") " pod="openshift-service-ca/service-ca-58598695f7-g27zm"
Jul 16 21:50:55.456435 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.456294    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:55.456435 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.456314    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gst86\" (UniqueName: \"kubernetes.io/projected/7b7695cb-b318-4e8c-a5bf-5dd7faa13223-kube-api-access-gst86\") pod \"service-ca-58598695f7-g27zm\" (UID: \"7b7695cb-b318-4e8c-a5bf-5dd7faa13223\") " pod="openshift-service-ca/service-ca-58598695f7-g27zm"
Jul 16 21:50:55.456435 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.456343    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7b7695cb-b318-4e8c-a5bf-5dd7faa13223-signing-cabundle\") pod \"service-ca-58598695f7-g27zm\" (UID: \"7b7695cb-b318-4e8c-a5bf-5dd7faa13223\") " pod="openshift-service-ca/service-ca-58598695f7-g27zm"
Jul 16 21:50:55.456435 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:55.456363    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:55.456435 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:55.456432    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls podName:8559cfa0-b3f8-4719-9e05-cef5ed2a5aec nodeName:}" failed. No retries permitted until 2026-07-16 21:50:59.456413536 +0000 UTC m=+44.053953747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r8jdg" (UID: "8559cfa0-b3f8-4719-9e05-cef5ed2a5aec") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:55.456435 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:55.456436    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:50:55.456695 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:55.456521    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm podName:8559cfa0-b3f8-4719-9e05-cef5ed2a5aec nodeName:}" failed. No retries permitted until 2026-07-16 21:50:59.456499778 +0000 UTC m=+44.054039992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r8jdg" (UID: "8559cfa0-b3f8-4719-9e05-cef5ed2a5aec") : configmap "kube-rbac-proxy" not found
Jul 16 21:50:55.556645 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.556613    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gst86\" (UniqueName: \"kubernetes.io/projected/7b7695cb-b318-4e8c-a5bf-5dd7faa13223-kube-api-access-gst86\") pod \"service-ca-58598695f7-g27zm\" (UID: \"7b7695cb-b318-4e8c-a5bf-5dd7faa13223\") " pod="openshift-service-ca/service-ca-58598695f7-g27zm"
Jul 16 21:50:55.557269 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.556654    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7b7695cb-b318-4e8c-a5bf-5dd7faa13223-signing-cabundle\") pod \"service-ca-58598695f7-g27zm\" (UID: \"7b7695cb-b318-4e8c-a5bf-5dd7faa13223\") " pod="openshift-service-ca/service-ca-58598695f7-g27zm"
Jul 16 21:50:55.557269 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.556875    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7b7695cb-b318-4e8c-a5bf-5dd7faa13223-signing-key\") pod \"service-ca-58598695f7-g27zm\" (UID: \"7b7695cb-b318-4e8c-a5bf-5dd7faa13223\") " pod="openshift-service-ca/service-ca-58598695f7-g27zm"
Jul 16 21:50:55.557391 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.557373    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7b7695cb-b318-4e8c-a5bf-5dd7faa13223-signing-cabundle\") pod \"service-ca-58598695f7-g27zm\" (UID: \"7b7695cb-b318-4e8c-a5bf-5dd7faa13223\") " pod="openshift-service-ca/service-ca-58598695f7-g27zm"
Jul 16 21:50:55.559988 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.559971    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7b7695cb-b318-4e8c-a5bf-5dd7faa13223-signing-key\") pod \"service-ca-58598695f7-g27zm\" (UID: \"7b7695cb-b318-4e8c-a5bf-5dd7faa13223\") " pod="openshift-service-ca/service-ca-58598695f7-g27zm"
Jul 16 21:50:55.565310 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.565290    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gst86\" (UniqueName: \"kubernetes.io/projected/7b7695cb-b318-4e8c-a5bf-5dd7faa13223-kube-api-access-gst86\") pod \"service-ca-58598695f7-g27zm\" (UID: \"7b7695cb-b318-4e8c-a5bf-5dd7faa13223\") " pod="openshift-service-ca/service-ca-58598695f7-g27zm"
Jul 16 21:50:55.666809 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.666761    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-g27zm"
Jul 16 21:50:55.809992 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:55.809962    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-g27zm"]
Jul 16 21:50:55.813005 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:50:55.812981    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7695cb_b318_4e8c_a5bf_5dd7faa13223.slice/crio-cf28a1c25ec2aacd42fce0e49d674bad9fa4bf801240bdb7730f969ace46545e WatchSource:0}: Error finding container cf28a1c25ec2aacd42fce0e49d674bad9fa4bf801240bdb7730f969ace46545e: Status 404 returned error can't find the container with id cf28a1c25ec2aacd42fce0e49d674bad9fa4bf801240bdb7730f969ace46545e
Jul 16 21:50:56.062240 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:56.062210    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zq96d" event={"ID":"ea5a4ca6-5adf-47f1-a1e4-4c59f554714d","Type":"ContainerStarted","Data":"9ccdc27f8c98ee61ce6e07b794ec7b923e66711ae8542de1b9ba85d086eea02b"}
Jul 16 21:50:56.063309 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:56.063288    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-g27zm" event={"ID":"7b7695cb-b318-4e8c-a5bf-5dd7faa13223","Type":"ContainerStarted","Data":"cf28a1c25ec2aacd42fce0e49d674bad9fa4bf801240bdb7730f969ace46545e"}
Jul 16 21:50:56.066233 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:56.066213    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4gx4" event={"ID":"65bfdb19-9745-4a5e-842a-05df0b73d7c0","Type":"ContainerStarted","Data":"b9c4322f93725e54963924d1b9661472d6dbbeddf52abbd0cfbf2bef6f011a67"}
Jul 16 21:50:56.081841 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:56.081800    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zq96d" podStartSLOduration=2.067069485 podStartE2EDuration="6.081787822s" totalImagesPullingTime="4.014718337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:56.081209233 +0000 UTC m=+40.678749463" watchObservedRunningTime="2026-07-16 21:50:56.081787822 +0000 UTC m=+40.679328053"
Jul 16 21:50:58.578595 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:58.578565    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:50:58.578924 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:58.578719    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:58.578924 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:58.578770    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls podName:f8a6451d-57f1-412b-9f81-e2903dfdff69 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:06.57875668 +0000 UTC m=+51.176296889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls") pod "dns-default-6b842" (UID: "f8a6451d-57f1-412b-9f81-e2903dfdff69") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:59.074733 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:59.074701    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-g27zm" event={"ID":"7b7695cb-b318-4e8c-a5bf-5dd7faa13223","Type":"ContainerStarted","Data":"35509670ce0056c81c52ca7f52f6842c2f16afd1b4bd2d5313d25caa40df4922"}
Jul 16 21:50:59.092388 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:59.092339    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-g27zm" podStartSLOduration=1.7945903730000001 podStartE2EDuration="4.092322975s" totalImagesPullingTime="2.297732602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:59.092145675 +0000 UTC m=+43.689685905" watchObservedRunningTime="2026-07-16 21:50:59.092322975 +0000 UTC m=+43.689863213"
Jul 16 21:50:59.092927 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:59.092896    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p4gx4" podStartSLOduration=11.677662182 podStartE2EDuration="43.09288964s" totalImagesPullingTime="31.415227458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:56.120807243 +0000 UTC m=+40.718347474" watchObservedRunningTime="2026-07-16 21:50:59.09288964 +0000 UTC m=+43.690429881"
Jul 16 21:50:59.486783 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:59.486739    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:59.486958 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:50:59.486824    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:50:59.486958 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:59.486867    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:59.486958 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:59.486940    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls podName:8559cfa0-b3f8-4719-9e05-cef5ed2a5aec nodeName:}" failed. No retries permitted until 2026-07-16 21:51:07.486919475 +0000 UTC m=+52.084459698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r8jdg" (UID: "8559cfa0-b3f8-4719-9e05-cef5ed2a5aec") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:59.486958 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:59.486946    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:50:59.487192 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:50:59.486994    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm podName:8559cfa0-b3f8-4719-9e05-cef5ed2a5aec nodeName:}" failed. No retries permitted until 2026-07-16 21:51:07.486979792 +0000 UTC m=+52.084520012 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r8jdg" (UID: "8559cfa0-b3f8-4719-9e05-cef5ed2a5aec") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:06.641475 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:06.641416    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:51:06.643594 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:06.643573    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a6451d-57f1-412b-9f81-e2903dfdff69-metrics-tls\") pod \"dns-default-6b842\" (UID: \"f8a6451d-57f1-412b-9f81-e2903dfdff69\") " pod="openshift-dns/dns-default-6b842"
Jul 16 21:51:06.694031 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:06.694003    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6b842"
Jul 16 21:51:06.829124 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:06.829093    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6b842"]
Jul 16 21:51:06.831916 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:51:06.831878    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a6451d_57f1_412b_9f81_e2903dfdff69.slice/crio-23642d754c4a2e94f5d0513bc3d8e173772ebf93c1c9371a1328d55a45c90708 WatchSource:0}: Error finding container 23642d754c4a2e94f5d0513bc3d8e173772ebf93c1c9371a1328d55a45c90708: Status 404 returned error can't find the container with id 23642d754c4a2e94f5d0513bc3d8e173772ebf93c1c9371a1328d55a45c90708
Jul 16 21:51:07.095773 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:07.095745    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6b842" event={"ID":"f8a6451d-57f1-412b-9f81-e2903dfdff69","Type":"ContainerStarted","Data":"23642d754c4a2e94f5d0513bc3d8e173772ebf93c1c9371a1328d55a45c90708"}
Jul 16 21:51:07.549003 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:07.548971    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:51:07.549186 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:07.549034    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:51:07.549186 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:51:07.549152    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:07.549304 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:51:07.549235    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls podName:8559cfa0-b3f8-4719-9e05-cef5ed2a5aec nodeName:}" failed. No retries permitted until 2026-07-16 21:51:23.549204109 +0000 UTC m=+68.146744318 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r8jdg" (UID: "8559cfa0-b3f8-4719-9e05-cef5ed2a5aec") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:07.549673 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:07.549646    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:51:09.102427 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:09.102398    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6b842" event={"ID":"f8a6451d-57f1-412b-9f81-e2903dfdff69","Type":"ContainerStarted","Data":"6bb376aad4778e21f48fa76f6e24c4b528a4550747daf335dc0fde87f2a9d0d6"}
Jul 16 21:51:10.106821 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:10.106788    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6b842" event={"ID":"f8a6451d-57f1-412b-9f81-e2903dfdff69","Type":"ContainerStarted","Data":"c881164cbd9a0c68bae12b664e13f83b06a937c380386c2e327033235e4f3376"}
Jul 16 21:51:10.107251 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:10.106907    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6b842"
Jul 16 21:51:10.126405 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:10.126345    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6b842" podStartSLOduration=18.103819276 podStartE2EDuration="20.126333805s" totalImagesPullingTime="2.022514529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:10.125510828 +0000 UTC m=+54.723051061" watchObservedRunningTime="2026-07-16 21:51:10.126333805 +0000 UTC m=+54.723874037"
Jul 16 21:51:17.040587 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:17.040561    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6hvb"
Jul 16 21:51:20.111591 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:20.111566    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6b842"
Jul 16 21:51:21.541968 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.541935    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:51:21.548994 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.548976    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:51:21.555125 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.555106    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2365d47-8e55-4776-b882-49eead286f8c-metrics-certs\") pod \"network-metrics-daemon-c2dgn\" (UID: \"b2365d47-8e55-4776-b882-49eead286f8c\") " pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:51:21.642528 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.642503    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:51:21.645994 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.645979    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:51:21.655093 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.655061    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbbaa953-ab0b-4594-a449-06fab72528f5-original-pull-secret\") pod \"global-pull-secret-syncer-m7rl6\" (UID: \"cbbaa953-ab0b-4594-a449-06fab72528f5\") " pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:51:21.743712 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.743690    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptz5\" (UniqueName: \"kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5\") pod \"network-check-target-s9mhc\" (UID: \"d5c93b6e-7ae4-4340-86bd-c10c6b804523\") " pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:51:21.746114 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.746099    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:51:21.757623 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.757607    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:51:21.767595 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.767567    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vptz5\" (UniqueName: \"kubernetes.io/projected/d5c93b6e-7ae4-4340-86bd-c10c6b804523-kube-api-access-vptz5\") pod \"network-check-target-s9mhc\" (UID: \"d5c93b6e-7ae4-4340-86bd-c10c6b804523\") " pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:51:21.777212 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.777192    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7rl6"
Jul 16 21:51:21.790270 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.790249    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fdwm\""
Jul 16 21:51:21.793333 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.793291    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wslpm\""
Jul 16 21:51:21.795309 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.795295    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2dgn"
Jul 16 21:51:21.800208 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.800190    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:51:21.920163 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.920137    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m7rl6"]
Jul 16 21:51:21.924668 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:51:21.924641    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbaa953_ab0b_4594_a449_06fab72528f5.slice/crio-7bcc3aa6bcc86b275361e65beb5af0125dd78f11926e1733601c632961f8ab70 WatchSource:0}: Error finding container 7bcc3aa6bcc86b275361e65beb5af0125dd78f11926e1733601c632961f8ab70: Status 404 returned error can't find the container with id 7bcc3aa6bcc86b275361e65beb5af0125dd78f11926e1733601c632961f8ab70
Jul 16 21:51:21.941152 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.941127    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s9mhc"]
Jul 16 21:51:21.943949 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:51:21.943926    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c93b6e_7ae4_4340_86bd_c10c6b804523.slice/crio-391804aaecf67687ec1c9377f62f42ae464590b450161f354431e75fc3fe3f6e WatchSource:0}: Error finding container 391804aaecf67687ec1c9377f62f42ae464590b450161f354431e75fc3fe3f6e: Status 404 returned error can't find the container with id 391804aaecf67687ec1c9377f62f42ae464590b450161f354431e75fc3fe3f6e
Jul 16 21:51:21.954398 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:21.954380    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c2dgn"]
Jul 16 21:51:21.957630 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:51:21.957608    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2365d47_8e55_4776_b882_49eead286f8c.slice/crio-453c838669ded1cee82b4a5036e91ae01eeef9261b58dc096e10033005f1d024 WatchSource:0}: Error finding container 453c838669ded1cee82b4a5036e91ae01eeef9261b58dc096e10033005f1d024: Status 404 returned error can't find the container with id 453c838669ded1cee82b4a5036e91ae01eeef9261b58dc096e10033005f1d024
Jul 16 21:51:22.135346 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:22.135283    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m7rl6" event={"ID":"cbbaa953-ab0b-4594-a449-06fab72528f5","Type":"ContainerStarted","Data":"7bcc3aa6bcc86b275361e65beb5af0125dd78f11926e1733601c632961f8ab70"}
Jul 16 21:51:22.136307 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:22.136287    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s9mhc" event={"ID":"d5c93b6e-7ae4-4340-86bd-c10c6b804523","Type":"ContainerStarted","Data":"391804aaecf67687ec1c9377f62f42ae464590b450161f354431e75fc3fe3f6e"}
Jul 16 21:51:22.140551 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:22.140523    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c2dgn" event={"ID":"b2365d47-8e55-4776-b882-49eead286f8c","Type":"ContainerStarted","Data":"453c838669ded1cee82b4a5036e91ae01eeef9261b58dc096e10033005f1d024"}
Jul 16 21:51:23.559222 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:23.559184    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:51:23.562061 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:23.562031    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8559cfa0-b3f8-4719-9e05-cef5ed2a5aec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8jdg\" (UID: \"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec\") " pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:51:23.815928 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:23.815842    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s89fr\""
Jul 16 21:51:23.823918 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:23.823887    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r8jdg"
Jul 16 21:51:23.981203 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:23.981168    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r8jdg"]
Jul 16 21:51:24.151555 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:24.151469    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c2dgn" event={"ID":"b2365d47-8e55-4776-b882-49eead286f8c","Type":"ContainerStarted","Data":"6a4b5f06224b61925c8c5e42ef5427205303fb2cf385764d97f9c6dd2f15653b"}
Jul 16 21:51:24.151555 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:24.151536    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c2dgn" event={"ID":"b2365d47-8e55-4776-b882-49eead286f8c","Type":"ContainerStarted","Data":"f3a10ae519f0cb8b18122686961c66f78a3dce6edab2b0240c1ed69b41977f3b"}
Jul 16 21:51:24.179686 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:24.179446    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c2dgn" podStartSLOduration=66.92864843 podStartE2EDuration="1m8.179425111s" totalImagesPullingTime="1.250776681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:24.177156456 +0000 UTC m=+68.774696691" watchObservedRunningTime="2026-07-16 21:51:24.179425111 +0000 UTC m=+68.776965348"
Jul 16 21:51:25.005561 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.005529    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc"]
Jul 16 21:51:25.025830 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.025808    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"]
Jul 16 21:51:25.025985 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.025966    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc"
Jul 16 21:51:25.032446 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.032202    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-k5pcq\""
Jul 16 21:51:25.033108 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.033039    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 21:51:25.035239 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.033468    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 16 21:51:25.036575 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.036516    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 21:51:25.040167 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.040141    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 21:51:25.042919 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:51:25.042892    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8559cfa0_b3f8_4719_9e05_cef5ed2a5aec.slice/crio-3de06811b2f6995aa6b9331d7e1b07fd757d4dc1046b2edcd9fe3d6dd2d376ad WatchSource:0}: Error finding container 3de06811b2f6995aa6b9331d7e1b07fd757d4dc1046b2edcd9fe3d6dd2d376ad: Status 404 returned error can't find the container with id 3de06811b2f6995aa6b9331d7e1b07fd757d4dc1046b2edcd9fe3d6dd2d376ad
Jul 16 21:51:25.047625 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.047609    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc"]
Jul 16 21:51:25.047700 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.047631    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"]
Jul 16 21:51:25.047735 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.047707    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.057202 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.057178    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 16 21:51:25.057310 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.057212    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 16 21:51:25.057310 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.057218    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 16 21:51:25.057310 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.057226    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 16 21:51:25.071519 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.071498    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1163cc8c-da61-411a-9c2f-54e831c1edad-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-684df9d488-lscbc\" (UID: \"1163cc8c-da61-411a-9c2f-54e831c1edad\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc"
Jul 16 21:51:25.071616 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.071532    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-hub\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.071616 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.071576    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.071616 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.071606    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.071752 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.071636    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-additional-service-ca\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.071752 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.071665    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cc99\" (UniqueName: \"kubernetes.io/projected/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-kube-api-access-4cc99\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.071752 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.071690    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smdp9\" (UniqueName: \"kubernetes.io/projected/1163cc8c-da61-411a-9c2f-54e831c1edad-kube-api-access-smdp9\") pod \"managed-serviceaccount-addon-agent-684df9d488-lscbc\" (UID: \"1163cc8c-da61-411a-9c2f-54e831c1edad\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc"
Jul 16 21:51:25.071850 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.071746    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-ca\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.156364 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.156322    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r8jdg" event={"ID":"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec","Type":"ContainerStarted","Data":"3de06811b2f6995aa6b9331d7e1b07fd757d4dc1046b2edcd9fe3d6dd2d376ad"}
Jul 16 21:51:25.172141 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.172108    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.172290 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.172157    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.172290 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.172187    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-additional-service-ca\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.172290 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.172218    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cc99\" (UniqueName: \"kubernetes.io/projected/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-kube-api-access-4cc99\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.172290 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.172255    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smdp9\" (UniqueName: \"kubernetes.io/projected/1163cc8c-da61-411a-9c2f-54e831c1edad-kube-api-access-smdp9\") pod \"managed-serviceaccount-addon-agent-684df9d488-lscbc\" (UID: \"1163cc8c-da61-411a-9c2f-54e831c1edad\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc"
Jul 16 21:51:25.172489 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.172297    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-ca\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.172489 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.172347    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1163cc8c-da61-411a-9c2f-54e831c1edad-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-684df9d488-lscbc\" (UID: \"1163cc8c-da61-411a-9c2f-54e831c1edad\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc"
Jul 16 21:51:25.172489 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.172373    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-hub\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.178034 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.173413    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-additional-service-ca\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.178034 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.175784    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.178034 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.176045    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1163cc8c-da61-411a-9c2f-54e831c1edad-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-684df9d488-lscbc\" (UID: \"1163cc8c-da61-411a-9c2f-54e831c1edad\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc"
Jul 16 21:51:25.178034 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.176412    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-ca\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.178034 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.177221    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.178590 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.178498    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-hub\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.209116 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.209076    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smdp9\" (UniqueName: \"kubernetes.io/projected/1163cc8c-da61-411a-9c2f-54e831c1edad-kube-api-access-smdp9\") pod \"managed-serviceaccount-addon-agent-684df9d488-lscbc\" (UID: \"1163cc8c-da61-411a-9c2f-54e831c1edad\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc"
Jul 16 21:51:25.222655 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.222635    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cc99\" (UniqueName: \"kubernetes.io/projected/a07b2c16-221d-4d25-9062-3e9d0b1fbf01-kube-api-access-4cc99\") pod \"cluster-proxy-proxy-agent-856bd57788-v9g5g\" (UID: \"a07b2c16-221d-4d25-9062-3e9d0b1fbf01\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:25.355374 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.355300    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc"
Jul 16 21:51:25.362509 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:25.362477    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"
Jul 16 21:51:27.068245 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:27.068220    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc"]
Jul 16 21:51:27.073487 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:51:27.073457    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1163cc8c_da61_411a_9c2f_54e831c1edad.slice/crio-0ffdf53d85cd9eb79195383bfa46e0c42e96e832a882e31b1454a8e283c7ff6d WatchSource:0}: Error finding container 0ffdf53d85cd9eb79195383bfa46e0c42e96e832a882e31b1454a8e283c7ff6d: Status 404 returned error can't find the container with id 0ffdf53d85cd9eb79195383bfa46e0c42e96e832a882e31b1454a8e283c7ff6d
Jul 16 21:51:27.087666 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:27.087640    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g"]
Jul 16 21:51:27.091747 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:51:27.091719    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda07b2c16_221d_4d25_9062_3e9d0b1fbf01.slice/crio-6dca6300daabd0978c6d09b20e9f3886468f2db72f384397df3b276278b358ef WatchSource:0}: Error finding container 6dca6300daabd0978c6d09b20e9f3886468f2db72f384397df3b276278b358ef: Status 404 returned error can't find the container with id 6dca6300daabd0978c6d09b20e9f3886468f2db72f384397df3b276278b358ef
Jul 16 21:51:27.165690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:27.165581    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s9mhc" event={"ID":"d5c93b6e-7ae4-4340-86bd-c10c6b804523","Type":"ContainerStarted","Data":"f5889c613bdc3ef27e74de1f8cd5df4db19d996cb24e9dbd6b723ccf4ef09cac"}
Jul 16 21:51:27.165690 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:27.165661    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:51:27.167642 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:27.167565    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g" event={"ID":"a07b2c16-221d-4d25-9062-3e9d0b1fbf01","Type":"ContainerStarted","Data":"6dca6300daabd0978c6d09b20e9f3886468f2db72f384397df3b276278b358ef"}
Jul 16 21:51:27.172056 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:27.172007    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m7rl6" event={"ID":"cbbaa953-ab0b-4594-a449-06fab72528f5","Type":"ContainerStarted","Data":"22ff5d44599c21376234cfc54bd91ece3623c359ccb34510e4a60c5764a03850"}
Jul 16 21:51:27.174296 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:27.174257    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc" event={"ID":"1163cc8c-da61-411a-9c2f-54e831c1edad","Type":"ContainerStarted","Data":"0ffdf53d85cd9eb79195383bfa46e0c42e96e832a882e31b1454a8e283c7ff6d"}
Jul 16 21:51:27.203011 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:27.202734    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s9mhc" podStartSLOduration=66.246350718 podStartE2EDuration="1m11.202715104s" totalImagesPullingTime="4.956364386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:27.185399942 +0000 UTC m=+71.782940175" watchObservedRunningTime="2026-07-16 21:51:27.202715104 +0000 UTC m=+71.800255338"
Jul 16 21:51:27.203246 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:27.203176    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m7rl6" podStartSLOduration=66.210693837 podStartE2EDuration="1m11.203163149s" totalImagesPullingTime="4.992469312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:27.201775526 +0000 UTC m=+71.799315759" watchObservedRunningTime="2026-07-16 21:51:27.203163149 +0000 UTC m=+71.800703382"
Jul 16 21:51:34.205819 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:34.205739    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g" event={"ID":"a07b2c16-221d-4d25-9062-3e9d0b1fbf01","Type":"ContainerStarted","Data":"db47ee0d4d9d491c09d0ff0e203e62c699005344b280d97c16d517888de0143c"}
Jul 16 21:51:34.205819 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:34.205792    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g" event={"ID":"a07b2c16-221d-4d25-9062-3e9d0b1fbf01","Type":"ContainerStarted","Data":"94cdac94bf763d4817a2f01c760dfabd1ebeed4e5a2e8b15b0e6f9c42b28ea0f"}
Jul 16 21:51:34.205819 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:34.205809    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g" event={"ID":"a07b2c16-221d-4d25-9062-3e9d0b1fbf01","Type":"ContainerStarted","Data":"51aef14e05e93d75391929833356bbd1e08c1ef71c96ad63b591794291c18a67"}
Jul 16 21:51:34.207196 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:34.207171    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc" event={"ID":"1163cc8c-da61-411a-9c2f-54e831c1edad","Type":"ContainerStarted","Data":"454359ecc95be9b65dac1b59c00c3f41b1b0e6b5a7d865a22ee36324dafd112f"}
Jul 16 21:51:34.208416 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:34.208397    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r8jdg" event={"ID":"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec","Type":"ContainerStarted","Data":"6f1bd375352e34e2b043e9fefab52c281a6557c64db35dcf36970623cf5ae7aa"}
Jul 16 21:51:34.226253 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:34.226202    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-856bd57788-v9g5g" podStartSLOduration=3.564374293 podStartE2EDuration="10.226185539s" totalImagesPullingTime="6.661811246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:34.225726949 +0000 UTC m=+78.823267182" watchObservedRunningTime="2026-07-16 21:51:34.226185539 +0000 UTC m=+78.823725782"
Jul 16 21:51:34.241299 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:34.241261    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684df9d488-lscbc" podStartSLOduration=3.887703454 podStartE2EDuration="10.241249465s" totalImagesPullingTime="6.353546011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:34.241063967 +0000 UTC m=+78.838604199" watchObservedRunningTime="2026-07-16 21:51:34.241249465 +0000 UTC m=+78.838789700"
Jul 16 21:51:38.229420 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:38.229284    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r8jdg" event={"ID":"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec","Type":"ContainerStarted","Data":"9bbbd9233133fb0293d3cd52c0d5a8ffd9f1950f3f18af7cb9076063314e4896"}
Jul 16 21:51:41.242771 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.242731    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r8jdg" event={"ID":"8559cfa0-b3f8-4719-9e05-cef5ed2a5aec","Type":"ContainerStarted","Data":"f163db5dcf18ba6d4d64aae161a04079234f6051614f2b33632fd324c2f95830"}
Jul 16 21:51:41.304104 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.304034    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r8jdg" podStartSLOduration=34.621103151 podStartE2EDuration="50.304013642s" totalImagesPullingTime="15.682910491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:41.301282293 +0000 UTC m=+85.898822527" watchObservedRunningTime="2026-07-16 21:51:41.304013642 +0000 UTC m=+85.901553873"
Jul 16 21:51:41.438511 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.438468    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-chw9t"]
Jul 16 21:51:41.441763 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.441741    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-phmhr"]
Jul 16 21:51:41.441911 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.441889    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.445019 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.444998    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.446641 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.446618    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 21:51:41.446641 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.446634    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 21:51:41.446815 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.446674    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 21:51:41.447139 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.447119    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 21:51:41.447139 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.447132    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bkh4j\""
Jul 16 21:51:41.447290 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.447162    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 21:51:41.447290 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.447165    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 21:51:41.448251 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.448230    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 16 21:51:41.448694 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.448660    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 16 21:51:41.448790 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.448727    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 16 21:51:41.448956 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.448937    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kkqvv\""
Jul 16 21:51:41.466536 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.466508    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-phmhr"]
Jul 16 21:51:41.492747 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.492711    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.492747 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.492751    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29cee3ea-c23f-454a-9ea1-168e571b88bf-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.493044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.492769    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-tls\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.493044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.492785    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddbaa431-1284-4b0c-8f63-585d594e0473-sys\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.493044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.492863    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9857s\" (UniqueName: \"kubernetes.io/projected/ddbaa431-1284-4b0c-8f63-585d594e0473-kube-api-access-9857s\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.493044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.492922    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ddbaa431-1284-4b0c-8f63-585d594e0473-root\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.493044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.492959    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-textfile\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.493044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.492999    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddbaa431-1284-4b0c-8f63-585d594e0473-metrics-client-ca\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.493044 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.493033    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29cee3ea-c23f-454a-9ea1-168e571b88bf-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.493322 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.493066    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.493322 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.493128    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-wtmp\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.493322 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.493156    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.493322 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.493191    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.493322 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.493224    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-accelerators-collector-config\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.493322 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.493249    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmsqr\" (UniqueName: \"kubernetes.io/projected/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-api-access-qmsqr\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.593754 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.593716    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.593754 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.593761    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-accelerators-collector-config\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.593999 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.593778    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmsqr\" (UniqueName: \"kubernetes.io/projected/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-api-access-qmsqr\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.593999 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.593812    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.593999 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.593836    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29cee3ea-c23f-454a-9ea1-168e571b88bf-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.593999 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.593864    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-tls\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.593999 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.593889    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddbaa431-1284-4b0c-8f63-585d594e0473-sys\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.593999 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.593912    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9857s\" (UniqueName: \"kubernetes.io/projected/ddbaa431-1284-4b0c-8f63-585d594e0473-kube-api-access-9857s\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.594330 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.594172    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddbaa431-1284-4b0c-8f63-585d594e0473-sys\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.594330 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.594311    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ddbaa431-1284-4b0c-8f63-585d594e0473-root\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.594433 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.594359    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-textfile\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.594433 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.594386    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddbaa431-1284-4b0c-8f63-585d594e0473-metrics-client-ca\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.594433 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.594410    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29cee3ea-c23f-454a-9ea1-168e571b88bf-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.594592 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.594439    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.594592 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.594482    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-wtmp\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.594592 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.594512    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.594592 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.594540    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-accelerators-collector-config\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.594779 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.594616    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29cee3ea-c23f-454a-9ea1-168e571b88bf-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.594779 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:51:41.594713    2570 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 16 21:51:41.594779 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:51:41.594771    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-state-metrics-tls podName:29cee3ea-c23f-454a-9ea1-168e571b88bf nodeName:}" failed. No retries permitted until 2026-07-16 21:51:42.094752135 +0000 UTC m=+86.692292348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-phmhr" (UID: "29cee3ea-c23f-454a-9ea1-168e571b88bf") : secret "kube-state-metrics-tls" not found
Jul 16 21:51:41.594946 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.594803    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.595047 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.595025    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddbaa431-1284-4b0c-8f63-585d594e0473-metrics-client-ca\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.595150 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.595040    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ddbaa431-1284-4b0c-8f63-585d594e0473-root\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.595150 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.595055    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-wtmp\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.595265 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.595151    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29cee3ea-c23f-454a-9ea1-168e571b88bf-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.595322 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.595284    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-textfile\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.596671 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.596646    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.596763 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.596727    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ddbaa431-1284-4b0c-8f63-585d594e0473-node-exporter-tls\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.597208 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.597190    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.612756 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.612728    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9857s\" (UniqueName: \"kubernetes.io/projected/ddbaa431-1284-4b0c-8f63-585d594e0473-kube-api-access-9857s\") pod \"node-exporter-chw9t\" (UID: \"ddbaa431-1284-4b0c-8f63-585d594e0473\") " pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.612936 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.612918    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmsqr\" (UniqueName: \"kubernetes.io/projected/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-api-access-qmsqr\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:41.754761 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:41.754681    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-chw9t"
Jul 16 21:51:41.766380 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:51:41.766348    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbaa431_1284_4b0c_8f63_585d594e0473.slice/crio-72ab0eacb1e0371cdc4acbba3e8420f554571f99b4148c5ab0699214f62b1402 WatchSource:0}: Error finding container 72ab0eacb1e0371cdc4acbba3e8420f554571f99b4148c5ab0699214f62b1402: Status 404 returned error can't find the container with id 72ab0eacb1e0371cdc4acbba3e8420f554571f99b4148c5ab0699214f62b1402
Jul 16 21:51:42.098871 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:42.098785    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:42.101136 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:42.101117    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29cee3ea-c23f-454a-9ea1-168e571b88bf-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-phmhr\" (UID: \"29cee3ea-c23f-454a-9ea1-168e571b88bf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:42.249004 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:42.248960    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chw9t" event={"ID":"ddbaa431-1284-4b0c-8f63-585d594e0473","Type":"ContainerStarted","Data":"72ab0eacb1e0371cdc4acbba3e8420f554571f99b4148c5ab0699214f62b1402"}
Jul 16 21:51:42.360166 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:42.360065    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr"
Jul 16 21:51:42.537202 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:42.537165    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-phmhr"]
Jul 16 21:51:42.610547 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:51:42.610469    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cee3ea_c23f_454a_9ea1_168e571b88bf.slice/crio-a439626f8bee4fa8f7347755ed2e847c4bbcfd96bb2c4dc9874d1ec87b2afd8d WatchSource:0}: Error finding container a439626f8bee4fa8f7347755ed2e847c4bbcfd96bb2c4dc9874d1ec87b2afd8d: Status 404 returned error can't find the container with id a439626f8bee4fa8f7347755ed2e847c4bbcfd96bb2c4dc9874d1ec87b2afd8d
Jul 16 21:51:43.255270 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:43.255225    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr" event={"ID":"29cee3ea-c23f-454a-9ea1-168e571b88bf","Type":"ContainerStarted","Data":"a439626f8bee4fa8f7347755ed2e847c4bbcfd96bb2c4dc9874d1ec87b2afd8d"}
Jul 16 21:51:43.257168 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:43.257127    2570 generic.go:356] "Generic (PLEG): container finished" podID="ddbaa431-1284-4b0c-8f63-585d594e0473" containerID="1d34f3de088a43368bbd9d3158192c0ee590f14559b6c4a90a2570384006d49c" exitCode=0
Jul 16 21:51:43.257168 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:43.257165    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chw9t" event={"ID":"ddbaa431-1284-4b0c-8f63-585d594e0473","Type":"ContainerDied","Data":"1d34f3de088a43368bbd9d3158192c0ee590f14559b6c4a90a2570384006d49c"}
Jul 16 21:51:44.264722 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:44.264681    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr" event={"ID":"29cee3ea-c23f-454a-9ea1-168e571b88bf","Type":"ContainerStarted","Data":"0cc4d7c587d4fc8512d538f027d6c2b5ad8283d51feed7d78c7ec520cd4b81b1"}
Jul 16 21:51:44.265144 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:44.264762    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr" event={"ID":"29cee3ea-c23f-454a-9ea1-168e571b88bf","Type":"ContainerStarted","Data":"46ab2d8f1f6e778e68ed8d014abf7e327cdb313768ddc998161279c4c6f6281b"}
Jul 16 21:51:44.265144 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:44.264789    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr" event={"ID":"29cee3ea-c23f-454a-9ea1-168e571b88bf","Type":"ContainerStarted","Data":"bebaba5907818e4dc1608f5226a52f30e8034ec34e841996e203f559e9ad8847"}
Jul 16 21:51:44.267554 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:44.267525    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chw9t" event={"ID":"ddbaa431-1284-4b0c-8f63-585d594e0473","Type":"ContainerStarted","Data":"aab7234d597a4243a431ca26f498297ad26e1f322f99d2bd52f27eff02abf183"}
Jul 16 21:51:44.267671 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:44.267579    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chw9t" event={"ID":"ddbaa431-1284-4b0c-8f63-585d594e0473","Type":"ContainerStarted","Data":"9d2e5151b3c43dbebbe9075550a6c15c4b27748353fcf52ad7f39a82e9f05d9e"}
Jul 16 21:51:44.296658 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:44.296594    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-phmhr" podStartSLOduration=2.017297195 podStartE2EDuration="3.296570237s" totalImagesPullingTime="1.279273042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:44.294583987 +0000 UTC m=+88.892124221" watchObservedRunningTime="2026-07-16 21:51:44.296570237 +0000 UTC m=+88.894110467"
Jul 16 21:51:44.316415 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:44.316355    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-chw9t" podStartSLOduration=2.366496807 podStartE2EDuration="3.316333844s" totalImagesPullingTime="886.267165ms" totalInitContainerRuntime="63.569872ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:44.314844159 +0000 UTC m=+88.912384391" watchObservedRunningTime="2026-07-16 21:51:44.316333844 +0000 UTC m=+88.913874077"
Jul 16 21:51:58.182783 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:51:58.182756    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s9mhc"
Jul 16 21:55:15.758110 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:55:15.758054    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 21:55:15.762767 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:55:15.762739    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 21:55:15.762908 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:55:15.762866    2570 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 21:56:28.261511 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.261428    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-5nqrm"]
Jul 16 21:56:28.264378 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.264361    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:28.267362 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.267341    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 16 21:56:28.268145 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.268125    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 16 21:56:28.268145 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.268139    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 16 21:56:28.268302 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.268192    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 16 21:56:28.268302 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.268265    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vksfj\""
Jul 16 21:56:28.270002 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.269982    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 16 21:56:28.282577 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.282534    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-5nqrm"]
Jul 16 21:56:28.424217 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.424180    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9e67215a-e453-4427-b4e5-aabc606b9a30-cabundle0\") pod \"keda-operator-77f648d694-5nqrm\" (UID: \"9e67215a-e453-4427-b4e5-aabc606b9a30\") " pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:28.424389 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.424239    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hp2m\" (UniqueName: \"kubernetes.io/projected/9e67215a-e453-4427-b4e5-aabc606b9a30-kube-api-access-6hp2m\") pod \"keda-operator-77f648d694-5nqrm\" (UID: \"9e67215a-e453-4427-b4e5-aabc606b9a30\") " pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:28.424389 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.424275    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9e67215a-e453-4427-b4e5-aabc606b9a30-certificates\") pod \"keda-operator-77f648d694-5nqrm\" (UID: \"9e67215a-e453-4427-b4e5-aabc606b9a30\") " pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:28.525424 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.525321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9e67215a-e453-4427-b4e5-aabc606b9a30-cabundle0\") pod \"keda-operator-77f648d694-5nqrm\" (UID: \"9e67215a-e453-4427-b4e5-aabc606b9a30\") " pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:28.525424 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.525407    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hp2m\" (UniqueName: \"kubernetes.io/projected/9e67215a-e453-4427-b4e5-aabc606b9a30-kube-api-access-6hp2m\") pod \"keda-operator-77f648d694-5nqrm\" (UID: \"9e67215a-e453-4427-b4e5-aabc606b9a30\") " pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:28.525615 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.525454    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9e67215a-e453-4427-b4e5-aabc606b9a30-certificates\") pod \"keda-operator-77f648d694-5nqrm\" (UID: \"9e67215a-e453-4427-b4e5-aabc606b9a30\") " pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:28.525615 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:56:28.525555    2570 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 16 21:56:28.525615 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:56:28.525575    2570 secret.go:281] references non-existent secret key: ca.crt
Jul 16 21:56:28.525615 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:56:28.525586    2570 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 16 21:56:28.525615 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:56:28.525602    2570 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-5nqrm: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 16 21:56:28.525796 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:56:28.525679    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e67215a-e453-4427-b4e5-aabc606b9a30-certificates podName:9e67215a-e453-4427-b4e5-aabc606b9a30 nodeName:}" failed. No retries permitted until 2026-07-16 21:56:29.025659519 +0000 UTC m=+373.623199731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9e67215a-e453-4427-b4e5-aabc606b9a30-certificates") pod "keda-operator-77f648d694-5nqrm" (UID: "9e67215a-e453-4427-b4e5-aabc606b9a30") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 16 21:56:28.526175 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.526156    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9e67215a-e453-4427-b4e5-aabc606b9a30-cabundle0\") pod \"keda-operator-77f648d694-5nqrm\" (UID: \"9e67215a-e453-4427-b4e5-aabc606b9a30\") " pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:28.535421 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:28.535396    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hp2m\" (UniqueName: \"kubernetes.io/projected/9e67215a-e453-4427-b4e5-aabc606b9a30-kube-api-access-6hp2m\") pod \"keda-operator-77f648d694-5nqrm\" (UID: \"9e67215a-e453-4427-b4e5-aabc606b9a30\") " pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:29.029327 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:29.029288    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9e67215a-e453-4427-b4e5-aabc606b9a30-certificates\") pod \"keda-operator-77f648d694-5nqrm\" (UID: \"9e67215a-e453-4427-b4e5-aabc606b9a30\") " pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:29.031788 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:29.031769    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9e67215a-e453-4427-b4e5-aabc606b9a30-certificates\") pod \"keda-operator-77f648d694-5nqrm\" (UID: \"9e67215a-e453-4427-b4e5-aabc606b9a30\") " pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:29.174487 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:29.174443    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:29.315700 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:29.315676    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-5nqrm"]
Jul 16 21:56:29.318223 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:56:29.318186    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e67215a_e453_4427_b4e5_aabc606b9a30.slice/crio-435cb8eb718bfc35e20d6af466b8082cb5929e251278b6984f6dc595bf6da6cb WatchSource:0}: Error finding container 435cb8eb718bfc35e20d6af466b8082cb5929e251278b6984f6dc595bf6da6cb: Status 404 returned error can't find the container with id 435cb8eb718bfc35e20d6af466b8082cb5929e251278b6984f6dc595bf6da6cb
Jul 16 21:56:29.319495 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:29.319476    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:56:30.216892 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:30.216808    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-5nqrm" event={"ID":"9e67215a-e453-4427-b4e5-aabc606b9a30","Type":"ContainerStarted","Data":"435cb8eb718bfc35e20d6af466b8082cb5929e251278b6984f6dc595bf6da6cb"}
Jul 16 21:56:33.228890 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:33.228855    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-5nqrm" event={"ID":"9e67215a-e453-4427-b4e5-aabc606b9a30","Type":"ContainerStarted","Data":"c5c226ab864d3200a0d3b721eb95c70df4d0b6648c5ff1902147cb547d72020e"}
Jul 16 21:56:33.229277 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:33.228987    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:56:33.248474 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:33.248424    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-5nqrm" podStartSLOduration=2.246588472 podStartE2EDuration="5.248410903s" totalImagesPullingTime="3.001822431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:33.247112543 +0000 UTC m=+377.844652776" watchObservedRunningTime="2026-07-16 21:56:33.248410903 +0000 UTC m=+377.845951280"
Jul 16 21:56:54.234569 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:56:54.234536    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-5nqrm"
Jul 16 21:57:37.319989 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.319955    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-cc4rq"]
Jul 16 21:57:37.323004 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.322987    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-cc4rq"
Jul 16 21:57:37.330519 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.330497    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 16 21:57:37.330931 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.330907    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-v2hr9\""
Jul 16 21:57:37.331049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.330964    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 16 21:57:37.331049 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.330970    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 16 21:57:37.353001 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.352972    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-cc4rq"]
Jul 16 21:57:37.376025 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.375996    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43a455ac-3b52-4c4b-b1e5-d02e507018cf-cert\") pod \"llmisvc-controller-manager-654c557856-cc4rq\" (UID: \"43a455ac-3b52-4c4b-b1e5-d02e507018cf\") " pod="kserve/llmisvc-controller-manager-654c557856-cc4rq"
Jul 16 21:57:37.376160 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.376050    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hfl7\" (UniqueName: \"kubernetes.io/projected/43a455ac-3b52-4c4b-b1e5-d02e507018cf-kube-api-access-2hfl7\") pod \"llmisvc-controller-manager-654c557856-cc4rq\" (UID: \"43a455ac-3b52-4c4b-b1e5-d02e507018cf\") " pod="kserve/llmisvc-controller-manager-654c557856-cc4rq"
Jul 16 21:57:37.381616 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.381591    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-tbnsr"]
Jul 16 21:57:37.384332 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.384316    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tbnsr"
Jul 16 21:57:37.388436 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.388414    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mqknh\""
Jul 16 21:57:37.388545 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.388439    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 16 21:57:37.414968 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.414944    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tbnsr"]
Jul 16 21:57:37.476635 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.476605    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hfl7\" (UniqueName: \"kubernetes.io/projected/43a455ac-3b52-4c4b-b1e5-d02e507018cf-kube-api-access-2hfl7\") pod \"llmisvc-controller-manager-654c557856-cc4rq\" (UID: \"43a455ac-3b52-4c4b-b1e5-d02e507018cf\") " pod="kserve/llmisvc-controller-manager-654c557856-cc4rq"
Jul 16 21:57:37.476753 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.476686    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/639bdce7-3c63-4d0f-81e6-b2193ad13f2b-data\") pod \"seaweedfs-64568bcd49-tbnsr\" (UID: \"639bdce7-3c63-4d0f-81e6-b2193ad13f2b\") " pod="kserve/seaweedfs-64568bcd49-tbnsr"
Jul 16 21:57:37.476753 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.476715    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2lz\" (UniqueName: \"kubernetes.io/projected/639bdce7-3c63-4d0f-81e6-b2193ad13f2b-kube-api-access-fw2lz\") pod \"seaweedfs-64568bcd49-tbnsr\" (UID: \"639bdce7-3c63-4d0f-81e6-b2193ad13f2b\") " pod="kserve/seaweedfs-64568bcd49-tbnsr"
Jul 16 21:57:37.476753 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.476747    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43a455ac-3b52-4c4b-b1e5-d02e507018cf-cert\") pod \"llmisvc-controller-manager-654c557856-cc4rq\" (UID: \"43a455ac-3b52-4c4b-b1e5-d02e507018cf\") " pod="kserve/llmisvc-controller-manager-654c557856-cc4rq"
Jul 16 21:57:37.476910 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:57:37.476851    2570 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 16 21:57:37.476949 ip-10-0-137-85 kubenswrapper[2570]: E0716 21:57:37.476926    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43a455ac-3b52-4c4b-b1e5-d02e507018cf-cert podName:43a455ac-3b52-4c4b-b1e5-d02e507018cf nodeName:}" failed. No retries permitted until 2026-07-16 21:57:37.976905409 +0000 UTC m=+442.574445626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/43a455ac-3b52-4c4b-b1e5-d02e507018cf-cert") pod "llmisvc-controller-manager-654c557856-cc4rq" (UID: "43a455ac-3b52-4c4b-b1e5-d02e507018cf") : secret "llmisvc-webhook-server-cert" not found
Jul 16 21:57:37.487942 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.487914    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hfl7\" (UniqueName: \"kubernetes.io/projected/43a455ac-3b52-4c4b-b1e5-d02e507018cf-kube-api-access-2hfl7\") pod \"llmisvc-controller-manager-654c557856-cc4rq\" (UID: \"43a455ac-3b52-4c4b-b1e5-d02e507018cf\") " pod="kserve/llmisvc-controller-manager-654c557856-cc4rq"
Jul 16 21:57:37.577868 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.577792    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/639bdce7-3c63-4d0f-81e6-b2193ad13f2b-data\") pod \"seaweedfs-64568bcd49-tbnsr\" (UID: \"639bdce7-3c63-4d0f-81e6-b2193ad13f2b\") " pod="kserve/seaweedfs-64568bcd49-tbnsr"
Jul 16 21:57:37.577868 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.577824    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2lz\" (UniqueName: \"kubernetes.io/projected/639bdce7-3c63-4d0f-81e6-b2193ad13f2b-kube-api-access-fw2lz\") pod \"seaweedfs-64568bcd49-tbnsr\" (UID: \"639bdce7-3c63-4d0f-81e6-b2193ad13f2b\") " pod="kserve/seaweedfs-64568bcd49-tbnsr"
Jul 16 21:57:37.578243 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.578220    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/639bdce7-3c63-4d0f-81e6-b2193ad13f2b-data\") pod \"seaweedfs-64568bcd49-tbnsr\" (UID: \"639bdce7-3c63-4d0f-81e6-b2193ad13f2b\") " pod="kserve/seaweedfs-64568bcd49-tbnsr"
Jul 16 21:57:37.671232 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.671201    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2lz\" (UniqueName: \"kubernetes.io/projected/639bdce7-3c63-4d0f-81e6-b2193ad13f2b-kube-api-access-fw2lz\") pod \"seaweedfs-64568bcd49-tbnsr\" (UID: \"639bdce7-3c63-4d0f-81e6-b2193ad13f2b\") " pod="kserve/seaweedfs-64568bcd49-tbnsr"
Jul 16 21:57:37.692791 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.692773    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tbnsr"
Jul 16 21:57:37.836764 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.836690    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tbnsr"]
Jul 16 21:57:37.839812 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:57:37.839786    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639bdce7_3c63_4d0f_81e6_b2193ad13f2b.slice/crio-9f3eb88d2b4ce360c734849bee12de38a0d6f68374711e93952ae0829e65c57b WatchSource:0}: Error finding container 9f3eb88d2b4ce360c734849bee12de38a0d6f68374711e93952ae0829e65c57b: Status 404 returned error can't find the container with id 9f3eb88d2b4ce360c734849bee12de38a0d6f68374711e93952ae0829e65c57b
Jul 16 21:57:37.981453 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.981409    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43a455ac-3b52-4c4b-b1e5-d02e507018cf-cert\") pod \"llmisvc-controller-manager-654c557856-cc4rq\" (UID: \"43a455ac-3b52-4c4b-b1e5-d02e507018cf\") " pod="kserve/llmisvc-controller-manager-654c557856-cc4rq"
Jul 16 21:57:37.983852 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:37.983833    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43a455ac-3b52-4c4b-b1e5-d02e507018cf-cert\") pod \"llmisvc-controller-manager-654c557856-cc4rq\" (UID: \"43a455ac-3b52-4c4b-b1e5-d02e507018cf\") " pod="kserve/llmisvc-controller-manager-654c557856-cc4rq"
Jul 16 21:57:38.233830 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:38.233804    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-cc4rq"
Jul 16 21:57:38.364379 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:38.364291    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-cc4rq"]
Jul 16 21:57:38.368537 ip-10-0-137-85 kubenswrapper[2570]: W0716 21:57:38.368507    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod43a455ac_3b52_4c4b_b1e5_d02e507018cf.slice/crio-f2b08a0965ba0385a23b0a7af4758c06840ef6a892b86219e20c6156766c3358 WatchSource:0}: Error finding container f2b08a0965ba0385a23b0a7af4758c06840ef6a892b86219e20c6156766c3358: Status 404 returned error can't find the container with id f2b08a0965ba0385a23b0a7af4758c06840ef6a892b86219e20c6156766c3358
Jul 16 21:57:38.450913 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:38.450861    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-cc4rq" event={"ID":"43a455ac-3b52-4c4b-b1e5-d02e507018cf","Type":"ContainerStarted","Data":"f2b08a0965ba0385a23b0a7af4758c06840ef6a892b86219e20c6156766c3358"}
Jul 16 21:57:38.452544 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:38.452510    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tbnsr" event={"ID":"639bdce7-3c63-4d0f-81e6-b2193ad13f2b","Type":"ContainerStarted","Data":"9f3eb88d2b4ce360c734849bee12de38a0d6f68374711e93952ae0829e65c57b"}
Jul 16 21:57:41.466188 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:41.466153    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-cc4rq" event={"ID":"43a455ac-3b52-4c4b-b1e5-d02e507018cf","Type":"ContainerStarted","Data":"abf4f900766d324b012107888a6b9fd95945628b9348dce31016ed0152d86003"}
Jul 16 21:57:41.466537 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:41.466313    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-cc4rq"
Jul 16 21:57:41.467481 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:41.467459    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tbnsr" event={"ID":"639bdce7-3c63-4d0f-81e6-b2193ad13f2b","Type":"ContainerStarted","Data":"5e0a090cf63169002bf65a5c2e2e6cb7942af1c98e196159564bf46cda791df3"}
Jul 16 21:57:41.467591 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:41.467575    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-tbnsr"
Jul 16 21:57:41.485569 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:41.485133    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-cc4rq" podStartSLOduration=1.492385408 podStartE2EDuration="4.485118424s" totalImagesPullingTime="2.992733016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:41.484417358 +0000 UTC m=+446.081957592" watchObservedRunningTime="2026-07-16 21:57:41.485118424 +0000 UTC m=+446.082658657"
Jul 16 21:57:41.501061 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:41.501019    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-tbnsr" podStartSLOduration=0.932398234 podStartE2EDuration="4.501005805s" totalImagesPullingTime="3.568607571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:41.500539582 +0000 UTC m=+446.098079814" watchObservedRunningTime="2026-07-16 21:57:41.501005805 +0000 UTC m=+446.098546036"
Jul 16 21:57:47.473278 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:57:47.473242    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-tbnsr"
Jul 16 21:58:12.473882 ip-10-0-137-85 kubenswrapper[2570]: I0716 21:58:12.473838    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-cc4rq"
Jul 16 22:00:15.789359 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:15.789327    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 22:00:15.789937 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:15.789764    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 22:00:40.458126 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.458031    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"]
Jul 16 22:00:40.461287 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.461271    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.464786 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.464765    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:00:40.464953 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.464927    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-5f43b-kube-rbac-proxy-sar-config\""
Jul 16 22:00:40.465029 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.464984    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xdcsv\""
Jul 16 22:00:40.465206 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.465136    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:00:40.465288 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.465219    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-5f43b-predictor-serving-cert\""
Jul 16 22:00:40.465288 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.465243    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 22:00:40.465391 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.465338    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:00:40.478276 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.478255    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"]
Jul 16 22:00:40.626376 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.626349    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7d5487f-c523-450c-b345-b026ba28dd9a-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.626550 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.626397    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7d5487f-c523-450c-b345-b026ba28dd9a-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.626550 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.626422    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7d5487f-c523-450c-b345-b026ba28dd9a-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.626550 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.626461    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7d5487f-c523-450c-b345-b026ba28dd9a-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.626550 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.626488    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9ms\" (UniqueName: \"kubernetes.io/projected/b7d5487f-c523-450c-b345-b026ba28dd9a-kube-api-access-hb9ms\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.726985 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.726962    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7d5487f-c523-450c-b345-b026ba28dd9a-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.727212 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.726999    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7d5487f-c523-450c-b345-b026ba28dd9a-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.727212 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.727026    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9ms\" (UniqueName: \"kubernetes.io/projected/b7d5487f-c523-450c-b345-b026ba28dd9a-kube-api-access-hb9ms\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.727212 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.727143    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7d5487f-c523-450c-b345-b026ba28dd9a-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.727387 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.727211    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7d5487f-c523-450c-b345-b026ba28dd9a-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.727387 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.727370    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7d5487f-c523-450c-b345-b026ba28dd9a-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.727792 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.727772    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7d5487f-c523-450c-b345-b026ba28dd9a-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.727842 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.727773    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7d5487f-c523-450c-b345-b026ba28dd9a-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.729542 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.729522    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7d5487f-c523-450c-b345-b026ba28dd9a-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.736221 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.736196    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9ms\" (UniqueName: \"kubernetes.io/projected/b7d5487f-c523-450c-b345-b026ba28dd9a-kube-api-access-hb9ms\") pod \"isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.771996 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.771973    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:00:40.902024 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:40.901992    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"]
Jul 16 22:00:40.904423 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:00:40.904397    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d5487f_c523_450c_b345_b026ba28dd9a.slice/crio-eb4a97856c113af03bbb3eda385289e186ca8809c680fe7228f3787529060c7d WatchSource:0}: Error finding container eb4a97856c113af03bbb3eda385289e186ca8809c680fe7228f3787529060c7d: Status 404 returned error can't find the container with id eb4a97856c113af03bbb3eda385289e186ca8809c680fe7228f3787529060c7d
Jul 16 22:00:41.087025 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:41.086952    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" event={"ID":"b7d5487f-c523-450c-b345-b026ba28dd9a","Type":"ContainerStarted","Data":"eb4a97856c113af03bbb3eda385289e186ca8809c680fe7228f3787529060c7d"}
Jul 16 22:00:45.102266 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:45.102227    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" event={"ID":"b7d5487f-c523-450c-b345-b026ba28dd9a","Type":"ContainerStarted","Data":"02cc8c46fc838ebec49daea05a0a0280155cfbd5da34bd293f652d3d64e1b07d"}
Jul 16 22:00:48.113300 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:48.113269    2570 generic.go:356] "Generic (PLEG): container finished" podID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerID="02cc8c46fc838ebec49daea05a0a0280155cfbd5da34bd293f652d3d64e1b07d" exitCode=0
Jul 16 22:00:48.113643 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:00:48.113343    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" event={"ID":"b7d5487f-c523-450c-b345-b026ba28dd9a","Type":"ContainerDied","Data":"02cc8c46fc838ebec49daea05a0a0280155cfbd5da34bd293f652d3d64e1b07d"}
Jul 16 22:01:02.176546 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:02.176507    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" event={"ID":"b7d5487f-c523-450c-b345-b026ba28dd9a","Type":"ContainerStarted","Data":"73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd"}
Jul 16 22:01:08.201663 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:08.201587    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" event={"ID":"b7d5487f-c523-450c-b345-b026ba28dd9a","Type":"ContainerStarted","Data":"306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e"}
Jul 16 22:01:12.223148 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:12.223109    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" event={"ID":"b7d5487f-c523-450c-b345-b026ba28dd9a","Type":"ContainerStarted","Data":"5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212"}
Jul 16 22:01:12.223554 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:12.223348    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:01:12.228363 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:12.228343    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:01:12.248557 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:12.248517    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podStartSLOduration=1.7607693549999999 podStartE2EDuration="32.248503918s" totalImagesPullingTime="26.783454024s" totalInitContainerRuntime="3.704280539s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:40 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:01:12.24642981 +0000 UTC m=+656.843970033" watchObservedRunningTime="2026-07-16 22:01:12.248503918 +0000 UTC m=+656.846044148"
Jul 16 22:01:13.227164 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:13.227123    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:01:13.227610 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:13.227202    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:01:13.227610 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:13.227523    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 16 22:01:13.227958 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:13.227934    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:01:14.230915 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:14.230869    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 16 22:01:14.231333 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:14.231149    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:01:24.231152 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:24.231108    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 16 22:01:24.231688 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:24.231603    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:01:34.231440 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:34.231393    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 16 22:01:34.231970 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:34.231945    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:01:44.231042 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:44.230990    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 16 22:01:44.231466 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:44.231439    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:01:54.231097 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:54.231057    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 16 22:01:54.231635 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:01:54.231600    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:02:04.231566 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:04.231510    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 16 22:02:04.232109 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:04.232065    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:02:14.231989 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:14.231898    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:02:14.232387 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:14.232149    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:02:25.549922 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.549886    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"]
Jul 16 22:02:25.550393 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.550163    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container" containerID="cri-o://73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd" gracePeriod=30
Jul 16 22:02:25.550393 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.550219    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kube-rbac-proxy" containerID="cri-o://306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e" gracePeriod=30
Jul 16 22:02:25.550393 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.550233    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent" containerID="cri-o://5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212" gracePeriod=30
Jul 16 22:02:25.694658 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.694629    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"]
Jul 16 22:02:25.698261 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.698234    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.700563 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.700541    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-a32f7-kube-rbac-proxy-sar-config\""
Jul 16 22:02:25.700673 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.700580    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-a32f7-predictor-serving-cert\""
Jul 16 22:02:25.713535 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.713514    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"]
Jul 16 22:02:25.769802 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.769772    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"]
Jul 16 22:02:25.772940 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.772926    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.777353 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.777333    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-a32f7-kube-rbac-proxy-sar-config\""
Jul 16 22:02:25.777458 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.777408    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-a32f7-predictor-serving-cert\""
Jul 16 22:02:25.785361 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.785336    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"]
Jul 16 22:02:25.885874 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.885809    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjgk\" (UniqueName: \"kubernetes.io/projected/738f7248-0233-4a30-bf21-f591d48e9a79-kube-api-access-9mjgk\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.885874 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.885840    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/738f7248-0233-4a30-bf21-f591d48e9a79-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.885874 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.885863    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a183ee2a-8930-4e81-9b29-13293cb21f21-cabundle-cert\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.886060 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.885891    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a183ee2a-8930-4e81-9b29-13293cb21f21-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.886060 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.885920    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/738f7248-0233-4a30-bf21-f591d48e9a79-proxy-tls\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.886060 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.885938    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a183ee2a-8930-4e81-9b29-13293cb21f21-proxy-tls\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.886060 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.885952    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/738f7248-0233-4a30-bf21-f591d48e9a79-cabundle-cert\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.886060 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.886028    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/738f7248-0233-4a30-bf21-f591d48e9a79-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.886246 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.886078    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxd2\" (UniqueName: \"kubernetes.io/projected/a183ee2a-8930-4e81-9b29-13293cb21f21-kube-api-access-bfxd2\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.886246 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.886108    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a183ee2a-8930-4e81-9b29-13293cb21f21-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.987276 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.987250    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxd2\" (UniqueName: \"kubernetes.io/projected/a183ee2a-8930-4e81-9b29-13293cb21f21-kube-api-access-bfxd2\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.987401 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.987281    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a183ee2a-8930-4e81-9b29-13293cb21f21-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.987401 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.987302    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjgk\" (UniqueName: \"kubernetes.io/projected/738f7248-0233-4a30-bf21-f591d48e9a79-kube-api-access-9mjgk\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.987494 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.987427    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/738f7248-0233-4a30-bf21-f591d48e9a79-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.987494 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.987479    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a183ee2a-8930-4e81-9b29-13293cb21f21-cabundle-cert\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.987620 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.987523    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a183ee2a-8930-4e81-9b29-13293cb21f21-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.987620 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.987561    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/738f7248-0233-4a30-bf21-f591d48e9a79-proxy-tls\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.987620 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.987591    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a183ee2a-8930-4e81-9b29-13293cb21f21-proxy-tls\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.987783 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.987669    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a183ee2a-8930-4e81-9b29-13293cb21f21-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.987783 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:02:25.987677    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-serving-cert: secret "isvc-xgboost-graph-raw-a32f7-predictor-serving-cert" not found
Jul 16 22:02:25.987783 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:02:25.987704    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-serving-cert: secret "isvc-sklearn-graph-raw-a32f7-predictor-serving-cert" not found
Jul 16 22:02:25.987783 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:02:25.987731    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a183ee2a-8930-4e81-9b29-13293cb21f21-proxy-tls podName:a183ee2a-8930-4e81-9b29-13293cb21f21 nodeName:}" failed. No retries permitted until 2026-07-16 22:02:26.487711074 +0000 UTC m=+731.085251301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a183ee2a-8930-4e81-9b29-13293cb21f21-proxy-tls") pod "isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" (UID: "a183ee2a-8930-4e81-9b29-13293cb21f21") : secret "isvc-xgboost-graph-raw-a32f7-predictor-serving-cert" not found
Jul 16 22:02:25.987783 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:02:25.987773    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738f7248-0233-4a30-bf21-f591d48e9a79-proxy-tls podName:738f7248-0233-4a30-bf21-f591d48e9a79 nodeName:}" failed. No retries permitted until 2026-07-16 22:02:26.487752712 +0000 UTC m=+731.085292925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/738f7248-0233-4a30-bf21-f591d48e9a79-proxy-tls") pod "isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" (UID: "738f7248-0233-4a30-bf21-f591d48e9a79") : secret "isvc-sklearn-graph-raw-a32f7-predictor-serving-cert" not found
Jul 16 22:02:25.988031 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.987796    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/738f7248-0233-4a30-bf21-f591d48e9a79-cabundle-cert\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.988031 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.987844    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/738f7248-0233-4a30-bf21-f591d48e9a79-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.988031 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.987863    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/738f7248-0233-4a30-bf21-f591d48e9a79-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.988237 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.988211    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a183ee2a-8930-4e81-9b29-13293cb21f21-cabundle-cert\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.988297 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.988255    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a183ee2a-8930-4e81-9b29-13293cb21f21-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:25.988454 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.988437    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/738f7248-0233-4a30-bf21-f591d48e9a79-cabundle-cert\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.988506 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.988446    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/738f7248-0233-4a30-bf21-f591d48e9a79-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.996420 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.996396    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjgk\" (UniqueName: \"kubernetes.io/projected/738f7248-0233-4a30-bf21-f591d48e9a79-kube-api-access-9mjgk\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:25.996509 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:25.996465    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxd2\" (UniqueName: \"kubernetes.io/projected/a183ee2a-8930-4e81-9b29-13293cb21f21-kube-api-access-bfxd2\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:26.482659 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:26.482628    2570 generic.go:356] "Generic (PLEG): container finished" podID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerID="306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e" exitCode=2
Jul 16 22:02:26.482809 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:26.482677    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" event={"ID":"b7d5487f-c523-450c-b345-b026ba28dd9a","Type":"ContainerDied","Data":"306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e"}
Jul 16 22:02:26.491914 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:26.491883    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/738f7248-0233-4a30-bf21-f591d48e9a79-proxy-tls\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:26.491914 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:26.491916    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a183ee2a-8930-4e81-9b29-13293cb21f21-proxy-tls\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:26.494374 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:26.494355    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/738f7248-0233-4a30-bf21-f591d48e9a79-proxy-tls\") pod \"isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:26.494476 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:26.494423    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a183ee2a-8930-4e81-9b29-13293cb21f21-proxy-tls\") pod \"isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:26.609380 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:26.609359    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:26.684541 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:26.684420    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:26.748633 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:26.748600    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"]
Jul 16 22:02:26.750278 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:02:26.750247    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738f7248_0233_4a30_bf21_f591d48e9a79.slice/crio-f332402759f700cca59764b8661b2bccff93e6e6ba7f6c86aa3b08ddb225e290 WatchSource:0}: Error finding container f332402759f700cca59764b8661b2bccff93e6e6ba7f6c86aa3b08ddb225e290: Status 404 returned error can't find the container with id f332402759f700cca59764b8661b2bccff93e6e6ba7f6c86aa3b08ddb225e290
Jul 16 22:02:26.831618 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:26.831581    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"]
Jul 16 22:02:26.834369 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:02:26.834345    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda183ee2a_8930_4e81_9b29_13293cb21f21.slice/crio-82bb4e07b9a9510f883ce6b7322f631264c79b43eb178d8efa89e8bf0ca857e0 WatchSource:0}: Error finding container 82bb4e07b9a9510f883ce6b7322f631264c79b43eb178d8efa89e8bf0ca857e0: Status 404 returned error can't find the container with id 82bb4e07b9a9510f883ce6b7322f631264c79b43eb178d8efa89e8bf0ca857e0
Jul 16 22:02:27.224187 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:27.224135    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.16:8643/healthz\": dial tcp 10.134.0.16:8643: connect: connection refused"
Jul 16 22:02:27.488635 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:27.488544    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" event={"ID":"a183ee2a-8930-4e81-9b29-13293cb21f21","Type":"ContainerStarted","Data":"9ff247378774cdca6539860c8d8f079fb0fe3a9e8709446dfd8d6c2ee9c7b6f2"}
Jul 16 22:02:27.488635 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:27.488612    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" event={"ID":"a183ee2a-8930-4e81-9b29-13293cb21f21","Type":"ContainerStarted","Data":"82bb4e07b9a9510f883ce6b7322f631264c79b43eb178d8efa89e8bf0ca857e0"}
Jul 16 22:02:27.490043 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:27.490020    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" event={"ID":"738f7248-0233-4a30-bf21-f591d48e9a79","Type":"ContainerStarted","Data":"9effe5268e7807255160fc30344344de40d81076ff14c6d024d347ef10f5df2c"}
Jul 16 22:02:27.490169 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:27.490063    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" event={"ID":"738f7248-0233-4a30-bf21-f591d48e9a79","Type":"ContainerStarted","Data":"f332402759f700cca59764b8661b2bccff93e6e6ba7f6c86aa3b08ddb225e290"}
Jul 16 22:02:30.504588 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:30.504552    2570 generic.go:356] "Generic (PLEG): container finished" podID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerID="73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd" exitCode=0
Jul 16 22:02:30.504984 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:30.504638    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" event={"ID":"b7d5487f-c523-450c-b345-b026ba28dd9a","Type":"ContainerDied","Data":"73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd"}
Jul 16 22:02:30.506162 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:30.506129    2570 generic.go:356] "Generic (PLEG): container finished" podID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerID="9ff247378774cdca6539860c8d8f079fb0fe3a9e8709446dfd8d6c2ee9c7b6f2" exitCode=0
Jul 16 22:02:30.506289 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:30.506173    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" event={"ID":"a183ee2a-8930-4e81-9b29-13293cb21f21","Type":"ContainerDied","Data":"9ff247378774cdca6539860c8d8f079fb0fe3a9e8709446dfd8d6c2ee9c7b6f2"}
Jul 16 22:02:30.507560 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:30.507541    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:02:31.513490 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:31.513449    2570 generic.go:356] "Generic (PLEG): container finished" podID="738f7248-0233-4a30-bf21-f591d48e9a79" containerID="9effe5268e7807255160fc30344344de40d81076ff14c6d024d347ef10f5df2c" exitCode=0
Jul 16 22:02:31.513909 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:31.513528    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" event={"ID":"738f7248-0233-4a30-bf21-f591d48e9a79","Type":"ContainerDied","Data":"9effe5268e7807255160fc30344344de40d81076ff14c6d024d347ef10f5df2c"}
Jul 16 22:02:32.223961 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:32.223914    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.16:8643/healthz\": dial tcp 10.134.0.16:8643: connect: connection refused"
Jul 16 22:02:33.525921 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:33.525886    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" event={"ID":"738f7248-0233-4a30-bf21-f591d48e9a79","Type":"ContainerStarted","Data":"9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919"}
Jul 16 22:02:33.526401 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:33.525948    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" event={"ID":"738f7248-0233-4a30-bf21-f591d48e9a79","Type":"ContainerStarted","Data":"be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560"}
Jul 16 22:02:33.526401 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:33.526263    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:33.526514 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:33.526420    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:33.527670 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:33.527639    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 16 22:02:33.560105 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:33.560038    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podStartSLOduration=4.823423604 podStartE2EDuration="8.560019463s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.736595859s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:02:33.559603644 +0000 UTC m=+738.157143876" watchObservedRunningTime="2026-07-16 22:02:33.560019463 +0000 UTC m=+738.157559697"
Jul 16 22:02:34.231750 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:34.231704    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 16 22:02:34.232149 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:34.232124    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:02:34.530334 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:34.530232    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 16 22:02:37.223855 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:37.223803    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.16:8643/healthz\": dial tcp 10.134.0.16:8643: connect: connection refused"
Jul 16 22:02:37.224432 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:37.223978    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:02:39.534656 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:39.534624    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:02:39.535152 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:39.535125    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 16 22:02:42.223788 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:42.223742    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.16:8643/healthz\": dial tcp 10.134.0.16:8643: connect: connection refused"
Jul 16 22:02:44.231312 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:44.231164    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 16 22:02:44.231826 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:44.231524    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:02:47.223957 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:47.223906    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.16:8643/healthz\": dial tcp 10.134.0.16:8643: connect: connection refused"
Jul 16 22:02:49.536012 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:49.535965    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 16 22:02:49.593298 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:49.593262    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" event={"ID":"a183ee2a-8930-4e81-9b29-13293cb21f21","Type":"ContainerStarted","Data":"5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d"}
Jul 16 22:02:49.593483 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:49.593337    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" event={"ID":"a183ee2a-8930-4e81-9b29-13293cb21f21","Type":"ContainerStarted","Data":"43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5"}
Jul 16 22:02:49.593598 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:49.593570    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:49.614618 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:49.614575    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podStartSLOduration=2.697402964 podStartE2EDuration="24.614561946s" totalImagesPullingTime="18.414823104s" totalInitContainerRuntime="3.502335878s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:02:49.612819883 +0000 UTC m=+754.210360115" watchObservedRunningTime="2026-07-16 22:02:49.614561946 +0000 UTC m=+754.212102240"
Jul 16 22:02:50.597804 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:50.597773    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:50.598842 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:50.598815    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 16 22:02:51.601221 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:51.601173    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 16 22:02:52.224095 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:52.224030    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.16:8643/healthz\": dial tcp 10.134.0.16:8643: connect: connection refused"
Jul 16 22:02:54.231494 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:54.231443    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 16 22:02:54.232024 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:54.231600    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:02:54.232024 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:54.231838    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:02:54.232024 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:54.231955    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:02:56.250758 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.250730    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:02:56.344738 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.344703    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7d5487f-c523-450c-b345-b026ba28dd9a-kserve-provision-location\") pod \"b7d5487f-c523-450c-b345-b026ba28dd9a\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") "
Jul 16 22:02:56.344920 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.344747    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb9ms\" (UniqueName: \"kubernetes.io/projected/b7d5487f-c523-450c-b345-b026ba28dd9a-kube-api-access-hb9ms\") pod \"b7d5487f-c523-450c-b345-b026ba28dd9a\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") "
Jul 16 22:02:56.344920 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.344772    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7d5487f-c523-450c-b345-b026ba28dd9a-proxy-tls\") pod \"b7d5487f-c523-450c-b345-b026ba28dd9a\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") "
Jul 16 22:02:56.344920 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.344890    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7d5487f-c523-450c-b345-b026ba28dd9a-kube-rbac-proxy-sar-config\") pod \"b7d5487f-c523-450c-b345-b026ba28dd9a\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") "
Jul 16 22:02:56.345107 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.344931    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7d5487f-c523-450c-b345-b026ba28dd9a-cabundle-cert\") pod \"b7d5487f-c523-450c-b345-b026ba28dd9a\" (UID: \"b7d5487f-c523-450c-b345-b026ba28dd9a\") "
Jul 16 22:02:56.345107 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.345036    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d5487f-c523-450c-b345-b026ba28dd9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b7d5487f-c523-450c-b345-b026ba28dd9a" (UID: "b7d5487f-c523-450c-b345-b026ba28dd9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:02:56.345227 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.345195    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7d5487f-c523-450c-b345-b026ba28dd9a-kserve-provision-location\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:02:56.345316 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.345294    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d5487f-c523-450c-b345-b026ba28dd9a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b7d5487f-c523-450c-b345-b026ba28dd9a" (UID: "b7d5487f-c523-450c-b345-b026ba28dd9a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:02:56.345367 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.345333    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d5487f-c523-450c-b345-b026ba28dd9a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b7d5487f-c523-450c-b345-b026ba28dd9a" (UID: "b7d5487f-c523-450c-b345-b026ba28dd9a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:02:56.347006 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.346985    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d5487f-c523-450c-b345-b026ba28dd9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7d5487f-c523-450c-b345-b026ba28dd9a" (UID: "b7d5487f-c523-450c-b345-b026ba28dd9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:02:56.347069 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.346987    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d5487f-c523-450c-b345-b026ba28dd9a-kube-api-access-hb9ms" (OuterVolumeSpecName: "kube-api-access-hb9ms") pod "b7d5487f-c523-450c-b345-b026ba28dd9a" (UID: "b7d5487f-c523-450c-b345-b026ba28dd9a"). InnerVolumeSpecName "kube-api-access-hb9ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:02:56.446543 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.446506    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7d5487f-c523-450c-b345-b026ba28dd9a-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:02:56.446543 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.446537    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7d5487f-c523-450c-b345-b026ba28dd9a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:02:56.446543 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.446547    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7d5487f-c523-450c-b345-b026ba28dd9a-cabundle-cert\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:02:56.446760 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.446556    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb9ms\" (UniqueName: \"kubernetes.io/projected/b7d5487f-c523-450c-b345-b026ba28dd9a-kube-api-access-hb9ms\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:02:56.605769 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.605696    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:02:56.606225 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.606195    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 16 22:02:56.622051 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.622021    2570 generic.go:356] "Generic (PLEG): container finished" podID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerID="5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212" exitCode=0
Jul 16 22:02:56.622159 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.622101    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" event={"ID":"b7d5487f-c523-450c-b345-b026ba28dd9a","Type":"ContainerDied","Data":"5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212"}
Jul 16 22:02:56.622159 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.622154    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc" event={"ID":"b7d5487f-c523-450c-b345-b026ba28dd9a","Type":"ContainerDied","Data":"eb4a97856c113af03bbb3eda385289e186ca8809c680fe7228f3787529060c7d"}
Jul 16 22:02:56.622229 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.622184    2570 scope.go:117] "RemoveContainer" containerID="5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212"
Jul 16 22:02:56.622229 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.622102    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"
Jul 16 22:02:56.631690 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.631668    2570 scope.go:117] "RemoveContainer" containerID="306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e"
Jul 16 22:02:56.638973 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.638957    2570 scope.go:117] "RemoveContainer" containerID="73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd"
Jul 16 22:02:56.645833 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.645817    2570 scope.go:117] "RemoveContainer" containerID="02cc8c46fc838ebec49daea05a0a0280155cfbd5da34bd293f652d3d64e1b07d"
Jul 16 22:02:56.649023 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.649002    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"]
Jul 16 22:02:56.653312 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.653296    2570 scope.go:117] "RemoveContainer" containerID="5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212"
Jul 16 22:02:56.653623 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:02:56.653597    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212\": container with ID starting with 5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212 not found: ID does not exist" containerID="5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212"
Jul 16 22:02:56.653730 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.653628    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212"} err="failed to get container status \"5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212\": rpc error: code = NotFound desc = could not find container \"5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212\": container with ID starting with 5bf95d78747fa982ca0c3389b312bc55670558d42b8a842586e58c8387f53212 not found: ID does not exist"
Jul 16 22:02:56.653730 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.653667    2570 scope.go:117] "RemoveContainer" containerID="306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e"
Jul 16 22:02:56.653945 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:02:56.653924    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e\": container with ID starting with 306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e not found: ID does not exist" containerID="306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e"
Jul 16 22:02:56.653997 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.653952    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e"} err="failed to get container status \"306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e\": rpc error: code = NotFound desc = could not find container \"306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e\": container with ID starting with 306b4ed95ef97c8f89e1549dfaf642620b1dceb2678120a98e0581fe3deeea1e not found: ID does not exist"
Jul 16 22:02:56.653997 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.653974    2570 scope.go:117] "RemoveContainer" containerID="73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd"
Jul 16 22:02:56.654294 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.654214    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc"]
Jul 16 22:02:56.654294 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:02:56.654278    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd\": container with ID starting with 73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd not found: ID does not exist" containerID="73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd"
Jul 16 22:02:56.654430 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.654301    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd"} err="failed to get container status \"73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd\": rpc error: code = NotFound desc = could not find container \"73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd\": container with ID starting with 73c3289289992ac277df03b067685a2b4053041d7b9278cf34e44df2e6c149dd not found: ID does not exist"
Jul 16 22:02:56.654430 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.654315    2570 scope.go:117] "RemoveContainer" containerID="02cc8c46fc838ebec49daea05a0a0280155cfbd5da34bd293f652d3d64e1b07d"
Jul 16 22:02:56.654529 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:02:56.654514    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02cc8c46fc838ebec49daea05a0a0280155cfbd5da34bd293f652d3d64e1b07d\": container with ID starting with 02cc8c46fc838ebec49daea05a0a0280155cfbd5da34bd293f652d3d64e1b07d not found: ID does not exist" containerID="02cc8c46fc838ebec49daea05a0a0280155cfbd5da34bd293f652d3d64e1b07d"
Jul 16 22:02:56.654566 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:56.654531    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02cc8c46fc838ebec49daea05a0a0280155cfbd5da34bd293f652d3d64e1b07d"} err="failed to get container status \"02cc8c46fc838ebec49daea05a0a0280155cfbd5da34bd293f652d3d64e1b07d\": rpc error: code = NotFound desc = could not find container \"02cc8c46fc838ebec49daea05a0a0280155cfbd5da34bd293f652d3d64e1b07d\": container with ID starting with 02cc8c46fc838ebec49daea05a0a0280155cfbd5da34bd293f652d3d64e1b07d not found: ID does not exist"
Jul 16 22:02:57.870661 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:57.870626    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" path="/var/lib/kubelet/pods/b7d5487f-c523-450c-b345-b026ba28dd9a/volumes"
Jul 16 22:02:59.536101 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:02:59.536043    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 16 22:03:06.606785 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:03:06.606743    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 16 22:03:09.535922 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:03:09.535883    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 16 22:03:16.606569 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:03:16.606524    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 16 22:03:19.535345 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:03:19.535302    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 16 22:03:26.607070 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:03:26.607023    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 16 22:03:29.535261 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:03:29.535220    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 16 22:03:36.606792 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:03:36.606754    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 16 22:03:39.535176 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:03:39.535071    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 16 22:03:46.607134 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:03:46.607071    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 16 22:03:49.536234 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:03:49.536200    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:03:56.607270 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:03:56.607234    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:04:15.887831 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.887791    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"]
Jul 16 22:04:15.888393 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.888023    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container" containerID="cri-o://be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560" gracePeriod=30
Jul 16 22:04:15.888393 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.888126    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kube-rbac-proxy" containerID="cri-o://9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919" gracePeriod=30
Jul 16 22:04:15.970220 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.970189    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"]
Jul 16 22:04:15.970583 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.970568    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="storage-initializer"
Jul 16 22:04:15.970647 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.970586    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="storage-initializer"
Jul 16 22:04:15.970647 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.970605    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent"
Jul 16 22:04:15.970647 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.970614    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent"
Jul 16 22:04:15.970647 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.970625    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kube-rbac-proxy"
Jul 16 22:04:15.970647 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.970634    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kube-rbac-proxy"
Jul 16 22:04:15.970910 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.970650    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container"
Jul 16 22:04:15.970910 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.970659    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container"
Jul 16 22:04:15.970910 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.970725    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kserve-container"
Jul 16 22:04:15.970910 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.970738    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="kube-rbac-proxy"
Jul 16 22:04:15.970910 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.970747    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d5487f-c523-450c-b345-b026ba28dd9a" containerName="agent"
Jul 16 22:04:15.974122 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.974077    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:15.976503 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.976482    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-7f37f-predictor-serving-cert\""
Jul 16 22:04:15.976722 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.976704    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-7f37f-kube-rbac-proxy-sar-config\""
Jul 16 22:04:15.990685 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:15.990661    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"]
Jul 16 22:04:16.019303 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.019276    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/020dc3fe-e94c-444d-b968-021fcd129218-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.019421 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.019319    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020dc3fe-e94c-444d-b968-021fcd129218-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.019421 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.019348    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020dc3fe-e94c-444d-b968-021fcd129218-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.019421 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.019406    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020dc3fe-e94c-444d-b968-021fcd129218-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.019540 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.019495    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv44n\" (UniqueName: \"kubernetes.io/projected/020dc3fe-e94c-444d-b968-021fcd129218-kube-api-access-pv44n\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.032503 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.032480    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"]
Jul 16 22:04:16.032758 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.032731    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container" containerID="cri-o://43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5" gracePeriod=30
Jul 16 22:04:16.032758 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.032742    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kube-rbac-proxy" containerID="cri-o://5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d" gracePeriod=30
Jul 16 22:04:16.074750 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.074728    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"]
Jul 16 22:04:16.078670 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.078653    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.081216 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.081192    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-7f37f-predictor-serving-cert\""
Jul 16 22:04:16.081851 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.081835    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-7f37f-kube-rbac-proxy-sar-config\""
Jul 16 22:04:16.090875 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.090855    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"]
Jul 16 22:04:16.120786 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.120758    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.121019 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.120998    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv44n\" (UniqueName: \"kubernetes.io/projected/020dc3fe-e94c-444d-b968-021fcd129218-kube-api-access-pv44n\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.121219 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.121040    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnpv6\" (UniqueName: \"kubernetes.io/projected/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kube-api-access-jnpv6\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.121288 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.121266    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/020dc3fe-e94c-444d-b968-021fcd129218-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.121349 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.121312    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020dc3fe-e94c-444d-b968-021fcd129218-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.121404 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.121350    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020dc3fe-e94c-444d-b968-021fcd129218-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.121404 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.121393    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020dc3fe-e94c-444d-b968-021fcd129218-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.121511 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.121443    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.121511 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.121468    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.121611 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.121523    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.121611 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:04:16.121533    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-7f37f-predictor-serving-cert" not found
Jul 16 22:04:16.121698 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:04:16.121630    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/020dc3fe-e94c-444d-b968-021fcd129218-proxy-tls podName:020dc3fe-e94c-444d-b968-021fcd129218 nodeName:}" failed. No retries permitted until 2026-07-16 22:04:16.621609275 +0000 UTC m=+841.219149484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/020dc3fe-e94c-444d-b968-021fcd129218-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" (UID: "020dc3fe-e94c-444d-b968-021fcd129218") : secret "isvc-sklearn-graph-raw-hpa-7f37f-predictor-serving-cert" not found
Jul 16 22:04:16.121744 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.121729    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020dc3fe-e94c-444d-b968-021fcd129218-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.121994 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.121971    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/020dc3fe-e94c-444d-b968-021fcd129218-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.122270 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.122247    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020dc3fe-e94c-444d-b968-021fcd129218-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.133346 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.133322    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv44n\" (UniqueName: \"kubernetes.io/projected/020dc3fe-e94c-444d-b968-021fcd129218-kube-api-access-pv44n\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.222756 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.222733    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.222878 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.222762    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.222878 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.222785    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.222878 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.222829    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.222878 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.222858    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnpv6\" (UniqueName: \"kubernetes.io/projected/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kube-api-access-jnpv6\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.223060 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:04:16.222878    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-7f37f-predictor-serving-cert" not found
Jul 16 22:04:16.223060 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:04:16.222944    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-proxy-tls podName:e9b499bf-dfe6-451b-b5c4-81aeb539eab7 nodeName:}" failed. No retries permitted until 2026-07-16 22:04:16.722926643 +0000 UTC m=+841.320466865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" (UID: "e9b499bf-dfe6-451b-b5c4-81aeb539eab7") : secret "isvc-xgboost-graph-raw-hpa-7f37f-predictor-serving-cert" not found
Jul 16 22:04:16.223191 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.223171    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.223639 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.223613    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.223724 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.223679    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.231816 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.231791    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnpv6\" (UniqueName: \"kubernetes.io/projected/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kube-api-access-jnpv6\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.601437 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.601388    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 16 22:04:16.606738 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.606707    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 16 22:04:16.625820 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.625788    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020dc3fe-e94c-444d-b968-021fcd129218-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.628155 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.628134    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020dc3fe-e94c-444d-b968-021fcd129218-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.726823 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.726794    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.729241 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.729213    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:16.885454 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.885388    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:16.936096 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.936051    2570 generic.go:356] "Generic (PLEG): container finished" podID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerID="5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d" exitCode=2
Jul 16 22:04:16.936495 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.936121    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" event={"ID":"a183ee2a-8930-4e81-9b29-13293cb21f21","Type":"ContainerDied","Data":"5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d"}
Jul 16 22:04:16.939798 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.939769    2570 generic.go:356] "Generic (PLEG): container finished" podID="738f7248-0233-4a30-bf21-f591d48e9a79" containerID="9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919" exitCode=2
Jul 16 22:04:16.939896 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.939826    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" event={"ID":"738f7248-0233-4a30-bf21-f591d48e9a79","Type":"ContainerDied","Data":"9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919"}
Jul 16 22:04:16.988428 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:16.988400    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:17.019313 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:17.019290    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"]
Jul 16 22:04:17.021152 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:04:17.021128    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020dc3fe_e94c_444d_b968_021fcd129218.slice/crio-8bf2406edeea362ae57f1bbd0bdb0de7bb9525a0de73ecd01e8dec7fbd1044f1 WatchSource:0}: Error finding container 8bf2406edeea362ae57f1bbd0bdb0de7bb9525a0de73ecd01e8dec7fbd1044f1: Status 404 returned error can't find the container with id 8bf2406edeea362ae57f1bbd0bdb0de7bb9525a0de73ecd01e8dec7fbd1044f1
Jul 16 22:04:17.123227 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:17.123163    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"]
Jul 16 22:04:17.125896 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:04:17.125867    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b499bf_dfe6_451b_b5c4_81aeb539eab7.slice/crio-dbb87caa55d0bf8ee447889840523307c6607fab52b67b1a30181348f2c1f376 WatchSource:0}: Error finding container dbb87caa55d0bf8ee447889840523307c6607fab52b67b1a30181348f2c1f376: Status 404 returned error can't find the container with id dbb87caa55d0bf8ee447889840523307c6607fab52b67b1a30181348f2c1f376
Jul 16 22:04:17.944635 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:17.944593    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" event={"ID":"e9b499bf-dfe6-451b-b5c4-81aeb539eab7","Type":"ContainerStarted","Data":"2976ea7b6361e8e4f4a9ced4c56a6a96dc4565e010139c1fc91cdf780a2b694f"}
Jul 16 22:04:17.945040 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:17.944658    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" event={"ID":"e9b499bf-dfe6-451b-b5c4-81aeb539eab7","Type":"ContainerStarted","Data":"dbb87caa55d0bf8ee447889840523307c6607fab52b67b1a30181348f2c1f376"}
Jul 16 22:04:17.946054 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:17.946032    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" event={"ID":"020dc3fe-e94c-444d-b968-021fcd129218","Type":"ContainerStarted","Data":"716788414c4265896561b093108a47d0cd1f7bd18a75b33308e5e61d791c5c49"}
Jul 16 22:04:17.946149 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:17.946093    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" event={"ID":"020dc3fe-e94c-444d-b968-021fcd129218","Type":"ContainerStarted","Data":"8bf2406edeea362ae57f1bbd0bdb0de7bb9525a0de73ecd01e8dec7fbd1044f1"}
Jul 16 22:04:19.291388 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.291366    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:04:19.348177 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.348150    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a183ee2a-8930-4e81-9b29-13293cb21f21-kube-rbac-proxy-sar-config\") pod \"a183ee2a-8930-4e81-9b29-13293cb21f21\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") "
Jul 16 22:04:19.348285 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.348200    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a183ee2a-8930-4e81-9b29-13293cb21f21-cabundle-cert\") pod \"a183ee2a-8930-4e81-9b29-13293cb21f21\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") "
Jul 16 22:04:19.348285 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.348235    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfxd2\" (UniqueName: \"kubernetes.io/projected/a183ee2a-8930-4e81-9b29-13293cb21f21-kube-api-access-bfxd2\") pod \"a183ee2a-8930-4e81-9b29-13293cb21f21\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") "
Jul 16 22:04:19.348285 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.348254    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a183ee2a-8930-4e81-9b29-13293cb21f21-proxy-tls\") pod \"a183ee2a-8930-4e81-9b29-13293cb21f21\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") "
Jul 16 22:04:19.348285 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.348276    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a183ee2a-8930-4e81-9b29-13293cb21f21-kserve-provision-location\") pod \"a183ee2a-8930-4e81-9b29-13293cb21f21\" (UID: \"a183ee2a-8930-4e81-9b29-13293cb21f21\") "
Jul 16 22:04:19.348529 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.348502    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a183ee2a-8930-4e81-9b29-13293cb21f21-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a183ee2a-8930-4e81-9b29-13293cb21f21" (UID: "a183ee2a-8930-4e81-9b29-13293cb21f21"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:04:19.348649 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.348621    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a183ee2a-8930-4e81-9b29-13293cb21f21-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a183ee2a-8930-4e81-9b29-13293cb21f21" (UID: "a183ee2a-8930-4e81-9b29-13293cb21f21"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:04:19.348649 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.348641    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a183ee2a-8930-4e81-9b29-13293cb21f21-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a183ee2a-8930-4e81-9b29-13293cb21f21" (UID: "a183ee2a-8930-4e81-9b29-13293cb21f21"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:04:19.350288 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.350261    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a183ee2a-8930-4e81-9b29-13293cb21f21-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a183ee2a-8930-4e81-9b29-13293cb21f21" (UID: "a183ee2a-8930-4e81-9b29-13293cb21f21"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:04:19.350385 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.350353    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a183ee2a-8930-4e81-9b29-13293cb21f21-kube-api-access-bfxd2" (OuterVolumeSpecName: "kube-api-access-bfxd2") pod "a183ee2a-8930-4e81-9b29-13293cb21f21" (UID: "a183ee2a-8930-4e81-9b29-13293cb21f21"). InnerVolumeSpecName "kube-api-access-bfxd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:04:19.448916 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.448858    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a183ee2a-8930-4e81-9b29-13293cb21f21-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:04:19.448916 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.448883    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a183ee2a-8930-4e81-9b29-13293cb21f21-cabundle-cert\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:04:19.448916 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.448898    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfxd2\" (UniqueName: \"kubernetes.io/projected/a183ee2a-8930-4e81-9b29-13293cb21f21-kube-api-access-bfxd2\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:04:19.448916 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.448911    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a183ee2a-8930-4e81-9b29-13293cb21f21-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:04:19.449121 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.448925    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a183ee2a-8930-4e81-9b29-13293cb21f21-kserve-provision-location\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:04:19.530655 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.530613    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.17:8643/healthz\": dial tcp 10.134.0.17:8643: connect: connection refused"
Jul 16 22:04:19.536011 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.535985    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 16 22:04:19.955231 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.955204    2570 generic.go:356] "Generic (PLEG): container finished" podID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerID="43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5" exitCode=0
Jul 16 22:04:19.955328 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.955239    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" event={"ID":"a183ee2a-8930-4e81-9b29-13293cb21f21","Type":"ContainerDied","Data":"43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5"}
Jul 16 22:04:19.955328 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.955267    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"
Jul 16 22:04:19.955328 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.955285    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz" event={"ID":"a183ee2a-8930-4e81-9b29-13293cb21f21","Type":"ContainerDied","Data":"82bb4e07b9a9510f883ce6b7322f631264c79b43eb178d8efa89e8bf0ca857e0"}
Jul 16 22:04:19.955328 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:19.955314    2570 scope.go:117] "RemoveContainer" containerID="5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d"
Jul 16 22:04:20.002554 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.002527    2570 scope.go:117] "RemoveContainer" containerID="43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5"
Jul 16 22:04:20.013451 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.013428    2570 scope.go:117] "RemoveContainer" containerID="9ff247378774cdca6539860c8d8f079fb0fe3a9e8709446dfd8d6c2ee9c7b6f2"
Jul 16 22:04:20.014847 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.014826    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"]
Jul 16 22:04:20.019343 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.019320    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz"]
Jul 16 22:04:20.021254 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.021232    2570 scope.go:117] "RemoveContainer" containerID="5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d"
Jul 16 22:04:20.021515 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:04:20.021495    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d\": container with ID starting with 5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d not found: ID does not exist" containerID="5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d"
Jul 16 22:04:20.021579 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.021525    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d"} err="failed to get container status \"5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d\": rpc error: code = NotFound desc = could not find container \"5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d\": container with ID starting with 5392d669c46d5cd9fa3d5aaaf368fceef16f99712de30ce1127e1249c361301d not found: ID does not exist"
Jul 16 22:04:20.021579 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.021549    2570 scope.go:117] "RemoveContainer" containerID="43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5"
Jul 16 22:04:20.021781 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:04:20.021761    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5\": container with ID starting with 43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5 not found: ID does not exist" containerID="43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5"
Jul 16 22:04:20.021847 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.021791    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5"} err="failed to get container status \"43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5\": rpc error: code = NotFound desc = could not find container \"43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5\": container with ID starting with 43ab8cb2c0a063b0689ab1c0574d2b4f45adb1004765a58c18f09a4ebd6f54e5 not found: ID does not exist"
Jul 16 22:04:20.021847 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.021813    2570 scope.go:117] "RemoveContainer" containerID="9ff247378774cdca6539860c8d8f079fb0fe3a9e8709446dfd8d6c2ee9c7b6f2"
Jul 16 22:04:20.022038 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:04:20.022022    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff247378774cdca6539860c8d8f079fb0fe3a9e8709446dfd8d6c2ee9c7b6f2\": container with ID starting with 9ff247378774cdca6539860c8d8f079fb0fe3a9e8709446dfd8d6c2ee9c7b6f2 not found: ID does not exist" containerID="9ff247378774cdca6539860c8d8f079fb0fe3a9e8709446dfd8d6c2ee9c7b6f2"
Jul 16 22:04:20.022078 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.022041    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff247378774cdca6539860c8d8f079fb0fe3a9e8709446dfd8d6c2ee9c7b6f2"} err="failed to get container status \"9ff247378774cdca6539860c8d8f079fb0fe3a9e8709446dfd8d6c2ee9c7b6f2\": rpc error: code = NotFound desc = could not find container \"9ff247378774cdca6539860c8d8f079fb0fe3a9e8709446dfd8d6c2ee9c7b6f2\": container with ID starting with 9ff247378774cdca6539860c8d8f079fb0fe3a9e8709446dfd8d6c2ee9c7b6f2 not found: ID does not exist"
Jul 16 22:04:20.119478 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.119451    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:04:20.154277 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.154256    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/738f7248-0233-4a30-bf21-f591d48e9a79-proxy-tls\") pod \"738f7248-0233-4a30-bf21-f591d48e9a79\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") "
Jul 16 22:04:20.154391 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.154298    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/738f7248-0233-4a30-bf21-f591d48e9a79-kserve-provision-location\") pod \"738f7248-0233-4a30-bf21-f591d48e9a79\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") "
Jul 16 22:04:20.154391 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.154315    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/738f7248-0233-4a30-bf21-f591d48e9a79-cabundle-cert\") pod \"738f7248-0233-4a30-bf21-f591d48e9a79\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") "
Jul 16 22:04:20.154391 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.154348    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mjgk\" (UniqueName: \"kubernetes.io/projected/738f7248-0233-4a30-bf21-f591d48e9a79-kube-api-access-9mjgk\") pod \"738f7248-0233-4a30-bf21-f591d48e9a79\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") "
Jul 16 22:04:20.154391 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.154376    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/738f7248-0233-4a30-bf21-f591d48e9a79-kube-rbac-proxy-sar-config\") pod \"738f7248-0233-4a30-bf21-f591d48e9a79\" (UID: \"738f7248-0233-4a30-bf21-f591d48e9a79\") "
Jul 16 22:04:20.154627 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.154603    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/738f7248-0233-4a30-bf21-f591d48e9a79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "738f7248-0233-4a30-bf21-f591d48e9a79" (UID: "738f7248-0233-4a30-bf21-f591d48e9a79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:04:20.154734 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.154711    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738f7248-0233-4a30-bf21-f591d48e9a79-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "738f7248-0233-4a30-bf21-f591d48e9a79" (UID: "738f7248-0233-4a30-bf21-f591d48e9a79"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:04:20.154794 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.154778    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738f7248-0233-4a30-bf21-f591d48e9a79-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "738f7248-0233-4a30-bf21-f591d48e9a79" (UID: "738f7248-0233-4a30-bf21-f591d48e9a79"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:04:20.156267 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.156241    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/738f7248-0233-4a30-bf21-f591d48e9a79-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "738f7248-0233-4a30-bf21-f591d48e9a79" (UID: "738f7248-0233-4a30-bf21-f591d48e9a79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:04:20.156409 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.156384    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738f7248-0233-4a30-bf21-f591d48e9a79-kube-api-access-9mjgk" (OuterVolumeSpecName: "kube-api-access-9mjgk") pod "738f7248-0233-4a30-bf21-f591d48e9a79" (UID: "738f7248-0233-4a30-bf21-f591d48e9a79"). InnerVolumeSpecName "kube-api-access-9mjgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:04:20.255632 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.255600    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mjgk\" (UniqueName: \"kubernetes.io/projected/738f7248-0233-4a30-bf21-f591d48e9a79-kube-api-access-9mjgk\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:04:20.255632 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.255631    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/738f7248-0233-4a30-bf21-f591d48e9a79-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:04:20.255632 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.255641    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/738f7248-0233-4a30-bf21-f591d48e9a79-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:04:20.255786 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.255652    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/738f7248-0233-4a30-bf21-f591d48e9a79-kserve-provision-location\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:04:20.255786 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.255660    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/738f7248-0233-4a30-bf21-f591d48e9a79-cabundle-cert\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:04:20.966847 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.966736    2570 generic.go:356] "Generic (PLEG): container finished" podID="738f7248-0233-4a30-bf21-f591d48e9a79" containerID="be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560" exitCode=0
Jul 16 22:04:20.966847 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.966837    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"
Jul 16 22:04:20.967360 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.966765    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" event={"ID":"738f7248-0233-4a30-bf21-f591d48e9a79","Type":"ContainerDied","Data":"be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560"}
Jul 16 22:04:20.967360 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.966966    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x" event={"ID":"738f7248-0233-4a30-bf21-f591d48e9a79","Type":"ContainerDied","Data":"f332402759f700cca59764b8661b2bccff93e6e6ba7f6c86aa3b08ddb225e290"}
Jul 16 22:04:20.967360 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.967014    2570 scope.go:117] "RemoveContainer" containerID="9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919"
Jul 16 22:04:20.968852 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.968826    2570 generic.go:356] "Generic (PLEG): container finished" podID="020dc3fe-e94c-444d-b968-021fcd129218" containerID="716788414c4265896561b093108a47d0cd1f7bd18a75b33308e5e61d791c5c49" exitCode=0
Jul 16 22:04:20.968988 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.968873    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" event={"ID":"020dc3fe-e94c-444d-b968-021fcd129218","Type":"ContainerDied","Data":"716788414c4265896561b093108a47d0cd1f7bd18a75b33308e5e61d791c5c49"}
Jul 16 22:04:20.975752 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.975714    2570 scope.go:117] "RemoveContainer" containerID="be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560"
Jul 16 22:04:20.983281 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.983259    2570 scope.go:117] "RemoveContainer" containerID="9effe5268e7807255160fc30344344de40d81076ff14c6d024d347ef10f5df2c"
Jul 16 22:04:20.993778 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.993755    2570 scope.go:117] "RemoveContainer" containerID="9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919"
Jul 16 22:04:20.994062 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:04:20.994042    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919\": container with ID starting with 9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919 not found: ID does not exist" containerID="9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919"
Jul 16 22:04:20.994173 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.994069    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919"} err="failed to get container status \"9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919\": rpc error: code = NotFound desc = could not find container \"9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919\": container with ID starting with 9557eef53b17210912d408bee41892bdb61d8101c673502a6f72592718593919 not found: ID does not exist"
Jul 16 22:04:20.994173 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.994100    2570 scope.go:117] "RemoveContainer" containerID="be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560"
Jul 16 22:04:20.994376 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:04:20.994359    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560\": container with ID starting with be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560 not found: ID does not exist" containerID="be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560"
Jul 16 22:04:20.994435 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.994381    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560"} err="failed to get container status \"be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560\": rpc error: code = NotFound desc = could not find container \"be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560\": container with ID starting with be60bf894b528d8fb315400172700326afa15ef3760344aee6a7836288ded560 not found: ID does not exist"
Jul 16 22:04:20.994435 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.994396    2570 scope.go:117] "RemoveContainer" containerID="9effe5268e7807255160fc30344344de40d81076ff14c6d024d347ef10f5df2c"
Jul 16 22:04:20.994626 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:04:20.994606    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9effe5268e7807255160fc30344344de40d81076ff14c6d024d347ef10f5df2c\": container with ID starting with 9effe5268e7807255160fc30344344de40d81076ff14c6d024d347ef10f5df2c not found: ID does not exist" containerID="9effe5268e7807255160fc30344344de40d81076ff14c6d024d347ef10f5df2c"
Jul 16 22:04:20.994680 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:20.994632    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9effe5268e7807255160fc30344344de40d81076ff14c6d024d347ef10f5df2c"} err="failed to get container status \"9effe5268e7807255160fc30344344de40d81076ff14c6d024d347ef10f5df2c\": rpc error: code = NotFound desc = could not find container \"9effe5268e7807255160fc30344344de40d81076ff14c6d024d347ef10f5df2c\": container with ID starting with 9effe5268e7807255160fc30344344de40d81076ff14c6d024d347ef10f5df2c not found: ID does not exist"
Jul 16 22:04:21.006931 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:21.006911    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"]
Jul 16 22:04:21.013120 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:21.013071    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x"]
Jul 16 22:04:21.870894 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:21.870859    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" path="/var/lib/kubelet/pods/738f7248-0233-4a30-bf21-f591d48e9a79/volumes"
Jul 16 22:04:21.871362 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:21.871348    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" path="/var/lib/kubelet/pods/a183ee2a-8930-4e81-9b29-13293cb21f21/volumes"
Jul 16 22:04:21.975347 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:21.975318    2570 generic.go:356] "Generic (PLEG): container finished" podID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerID="2976ea7b6361e8e4f4a9ced4c56a6a96dc4565e010139c1fc91cdf780a2b694f" exitCode=0
Jul 16 22:04:21.975730 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:21.975391    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" event={"ID":"e9b499bf-dfe6-451b-b5c4-81aeb539eab7","Type":"ContainerDied","Data":"2976ea7b6361e8e4f4a9ced4c56a6a96dc4565e010139c1fc91cdf780a2b694f"}
Jul 16 22:04:21.977410 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:21.977390    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" event={"ID":"020dc3fe-e94c-444d-b968-021fcd129218","Type":"ContainerStarted","Data":"343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72"}
Jul 16 22:04:21.977509 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:21.977446    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" event={"ID":"020dc3fe-e94c-444d-b968-021fcd129218","Type":"ContainerStarted","Data":"a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01"}
Jul 16 22:04:21.977725 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:21.977704    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:21.977830 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:21.977779    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:21.979109 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:21.979067    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 16 22:04:22.017029 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:22.016987    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podStartSLOduration=3.393527413 podStartE2EDuration="7.016973316s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.623445903s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:04:22.015057417 +0000 UTC m=+846.612597652" watchObservedRunningTime="2026-07-16 22:04:22.016973316 +0000 UTC m=+846.614513548"
Jul 16 22:04:22.983159 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:22.983119    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" event={"ID":"e9b499bf-dfe6-451b-b5c4-81aeb539eab7","Type":"ContainerStarted","Data":"bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7"}
Jul 16 22:04:22.983574 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:22.983192    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" event={"ID":"e9b499bf-dfe6-451b-b5c4-81aeb539eab7","Type":"ContainerStarted","Data":"60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed"}
Jul 16 22:04:22.983574 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:22.983519    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 16 22:04:23.005869 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:23.005812    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" podStartSLOduration=3.131230222 podStartE2EDuration="7.005793273s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.874563051s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:04:23.003136414 +0000 UTC m=+847.600676647" watchObservedRunningTime="2026-07-16 22:04:23.005793273 +0000 UTC m=+847.603333506"
Jul 16 22:04:27.984076 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:27.984040    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:27.984537 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:27.984516    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:27.985397 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:27.985362    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 16 22:04:27.988394 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:27.988371    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:04:27.988825 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:27.988795    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 16 22:04:27.988825 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:27.988817    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:04:28.001075 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:28.001054    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 16 22:04:29.004620 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:29.004582    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 16 22:04:37.989806 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:37.989754    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 16 22:04:39.004961 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:39.004912    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 16 22:04:47.988962 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:47.988911    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 16 22:04:49.005442 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:49.005398    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 16 22:04:57.988774 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:57.988734    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 16 22:04:59.004852 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:04:59.004810    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 16 22:05:07.989545 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:05:07.989501    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 16 22:05:09.005207 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:05:09.005169    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 16 22:05:15.812047 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:05:15.812012    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 22:05:15.812715 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:05:15.812697    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 22:05:17.989728 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:05:17.989682    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 16 22:05:19.005279 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:05:19.005250    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:05:27.988916 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:05:27.988874    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 16 22:05:37.989560 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:05:37.989529    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:06:06.310584 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.310553    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"]
Jul 16 22:06:06.313014 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.310789    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container" containerID="cri-o://a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01" gracePeriod=30
Jul 16 22:06:06.313014 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.310837    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kube-rbac-proxy" containerID="cri-o://343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72" gracePeriod=30
Jul 16 22:06:06.362289 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362260    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"]
Jul 16 22:06:06.362556 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362544    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container"
Jul 16 22:06:06.362599 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362559    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container"
Jul 16 22:06:06.362599 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362574    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kube-rbac-proxy"
Jul 16 22:06:06.362599 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362585    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kube-rbac-proxy"
Jul 16 22:06:06.362599 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362596    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container"
Jul 16 22:06:06.362716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362601    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container"
Jul 16 22:06:06.362716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362608    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="storage-initializer"
Jul 16 22:06:06.362716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362613    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="storage-initializer"
Jul 16 22:06:06.362716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362620    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kube-rbac-proxy"
Jul 16 22:06:06.362716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362625    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kube-rbac-proxy"
Jul 16 22:06:06.362716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362633    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="storage-initializer"
Jul 16 22:06:06.362716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362638    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="storage-initializer"
Jul 16 22:06:06.362716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362696    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kube-rbac-proxy"
Jul 16 22:06:06.362716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362705    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kserve-container"
Jul 16 22:06:06.362716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362711    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a183ee2a-8930-4e81-9b29-13293cb21f21" containerName="kube-rbac-proxy"
Jul 16 22:06:06.362716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.362718    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="738f7248-0233-4a30-bf21-f591d48e9a79" containerName="kserve-container"
Jul 16 22:06:06.365598 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.365581    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:06.368121 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.368101    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-164a6-kube-rbac-proxy-sar-config\""
Jul 16 22:06:06.368206 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.368160    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-164a6-predictor-serving-cert\""
Jul 16 22:06:06.377068 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.377049    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"]
Jul 16 22:06:06.479884 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.479855    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"]
Jul 16 22:06:06.480136 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.480108    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kserve-container" containerID="cri-o://60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed" gracePeriod=30
Jul 16 22:06:06.480214 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.480153    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kube-rbac-proxy" containerID="cri-o://bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7" gracePeriod=30
Jul 16 22:06:06.498128 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.498103    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d3de467-678e-4ac6-a6ea-55c2192dcae4-proxy-tls\") pod \"message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") " pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:06.498212 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.498142    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d3de467-678e-4ac6-a6ea-55c2192dcae4-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") " pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:06.498254 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.498218    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjmnt\" (UniqueName: \"kubernetes.io/projected/5d3de467-678e-4ac6-a6ea-55c2192dcae4-kube-api-access-mjmnt\") pod \"message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") " pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:06.599034 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.599010    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmnt\" (UniqueName: \"kubernetes.io/projected/5d3de467-678e-4ac6-a6ea-55c2192dcae4-kube-api-access-mjmnt\") pod \"message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") " pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:06.599127 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.599060    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d3de467-678e-4ac6-a6ea-55c2192dcae4-proxy-tls\") pod \"message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") " pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:06.599127 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.599111    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d3de467-678e-4ac6-a6ea-55c2192dcae4-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") " pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:06.599223 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:06:06.599171    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-serving-cert: secret "message-dumper-raw-164a6-predictor-serving-cert" not found
Jul 16 22:06:06.599275 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:06:06.599235    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3de467-678e-4ac6-a6ea-55c2192dcae4-proxy-tls podName:5d3de467-678e-4ac6-a6ea-55c2192dcae4 nodeName:}" failed. No retries permitted until 2026-07-16 22:06:07.099216133 +0000 UTC m=+951.696756354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5d3de467-678e-4ac6-a6ea-55c2192dcae4-proxy-tls") pod "message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr" (UID: "5d3de467-678e-4ac6-a6ea-55c2192dcae4") : secret "message-dumper-raw-164a6-predictor-serving-cert" not found
Jul 16 22:06:06.599699 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.599682    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d3de467-678e-4ac6-a6ea-55c2192dcae4-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") " pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:06.607881 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:06.607857    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjmnt\" (UniqueName: \"kubernetes.io/projected/5d3de467-678e-4ac6-a6ea-55c2192dcae4-kube-api-access-mjmnt\") pod \"message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") " pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:07.102383 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:07.102353    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d3de467-678e-4ac6-a6ea-55c2192dcae4-proxy-tls\") pod \"message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") " pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:07.104716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:07.104692    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d3de467-678e-4ac6-a6ea-55c2192dcae4-proxy-tls\") pod \"message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") " pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:07.278301 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:07.278274    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:07.363859 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:07.363822    2570 generic.go:356] "Generic (PLEG): container finished" podID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerID="bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7" exitCode=2
Jul 16 22:06:07.364244 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:07.363903    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" event={"ID":"e9b499bf-dfe6-451b-b5c4-81aeb539eab7","Type":"ContainerDied","Data":"bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7"}
Jul 16 22:06:07.366471 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:07.366427    2570 generic.go:356] "Generic (PLEG): container finished" podID="020dc3fe-e94c-444d-b968-021fcd129218" containerID="343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72" exitCode=2
Jul 16 22:06:07.366586 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:07.366497    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" event={"ID":"020dc3fe-e94c-444d-b968-021fcd129218","Type":"ContainerDied","Data":"343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72"}
Jul 16 22:06:07.418986 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:07.418964    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"]
Jul 16 22:06:07.420552 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:06:07.420527    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3de467_678e_4ac6_a6ea_55c2192dcae4.slice/crio-ac9e91e454c2d23ea48e56c1239ba59685465b2e619b22a1cb77b14cac184532 WatchSource:0}: Error finding container ac9e91e454c2d23ea48e56c1239ba59685465b2e619b22a1cb77b14cac184532: Status 404 returned error can't find the container with id ac9e91e454c2d23ea48e56c1239ba59685465b2e619b22a1cb77b14cac184532
Jul 16 22:06:07.984078 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:07.984036    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 16 22:06:07.989480 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:07.989455    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 16 22:06:07.989583 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:07.989519    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 16 22:06:08.371458 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:08.371377    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr" event={"ID":"5d3de467-678e-4ac6-a6ea-55c2192dcae4","Type":"ContainerStarted","Data":"ac9e91e454c2d23ea48e56c1239ba59685465b2e619b22a1cb77b14cac184532"}
Jul 16 22:06:09.004884 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:09.004849    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 16 22:06:09.377033 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:09.376993    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr" event={"ID":"5d3de467-678e-4ac6-a6ea-55c2192dcae4","Type":"ContainerStarted","Data":"ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5"}
Jul 16 22:06:09.377660 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:09.377056    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr" event={"ID":"5d3de467-678e-4ac6-a6ea-55c2192dcae4","Type":"ContainerStarted","Data":"49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174"}
Jul 16 22:06:09.377660 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:09.377110    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:09.411701 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:09.411651    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr" podStartSLOduration=2.049699852 podStartE2EDuration="3.411633473s" totalImagesPullingTime="1.361933621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:06:09.407470929 +0000 UTC m=+954.005011161" watchObservedRunningTime="2026-07-16 22:06:09.411633473 +0000 UTC m=+954.009173705"
Jul 16 22:06:10.380960 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.380915    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:10.382598 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.382578    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:10.514793 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.514773    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:06:10.631891 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.631806    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kserve-provision-location\") pod \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") "
Jul 16 22:06:10.631891 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.631844    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnpv6\" (UniqueName: \"kubernetes.io/projected/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kube-api-access-jnpv6\") pod \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") "
Jul 16 22:06:10.631891 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.631863    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-cabundle-cert\") pod \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") "
Jul 16 22:06:10.631891 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.631886    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-proxy-tls\") pod \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") "
Jul 16 22:06:10.632202 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.632003    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kube-rbac-proxy-sar-config\") pod \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\" (UID: \"e9b499bf-dfe6-451b-b5c4-81aeb539eab7\") "
Jul 16 22:06:10.632244 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.632208    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9b499bf-dfe6-451b-b5c4-81aeb539eab7" (UID: "e9b499bf-dfe6-451b-b5c4-81aeb539eab7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:06:10.632344 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.632238    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e9b499bf-dfe6-451b-b5c4-81aeb539eab7" (UID: "e9b499bf-dfe6-451b-b5c4-81aeb539eab7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:06:10.632401 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.632373    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kserve-provision-location\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:06:10.632401 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.632370    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e9b499bf-dfe6-451b-b5c4-81aeb539eab7" (UID: "e9b499bf-dfe6-451b-b5c4-81aeb539eab7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:06:10.633942 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.633918    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9b499bf-dfe6-451b-b5c4-81aeb539eab7" (UID: "e9b499bf-dfe6-451b-b5c4-81aeb539eab7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:06:10.633942 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.633919    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kube-api-access-jnpv6" (OuterVolumeSpecName: "kube-api-access-jnpv6") pod "e9b499bf-dfe6-451b-b5c4-81aeb539eab7" (UID: "e9b499bf-dfe6-451b-b5c4-81aeb539eab7"). InnerVolumeSpecName "kube-api-access-jnpv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:06:10.733357 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.733335    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:06:10.733357 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.733357    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:06:10.733492 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.733366    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnpv6\" (UniqueName: \"kubernetes.io/projected/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-kube-api-access-jnpv6\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:06:10.733492 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:10.733376    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9b499bf-dfe6-451b-b5c4-81aeb539eab7-cabundle-cert\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:06:11.244673 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.244646    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:06:11.338206 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.338179    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020dc3fe-e94c-444d-b968-021fcd129218-proxy-tls\") pod \"020dc3fe-e94c-444d-b968-021fcd129218\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") "
Jul 16 22:06:11.338301 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.338223    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv44n\" (UniqueName: \"kubernetes.io/projected/020dc3fe-e94c-444d-b968-021fcd129218-kube-api-access-pv44n\") pod \"020dc3fe-e94c-444d-b968-021fcd129218\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") "
Jul 16 22:06:11.338301 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.338248    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020dc3fe-e94c-444d-b968-021fcd129218-cabundle-cert\") pod \"020dc3fe-e94c-444d-b968-021fcd129218\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") "
Jul 16 22:06:11.338301 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.338281    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020dc3fe-e94c-444d-b968-021fcd129218-kserve-provision-location\") pod \"020dc3fe-e94c-444d-b968-021fcd129218\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") "
Jul 16 22:06:11.338422 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.338309    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/020dc3fe-e94c-444d-b968-021fcd129218-kube-rbac-proxy-sar-config\") pod \"020dc3fe-e94c-444d-b968-021fcd129218\" (UID: \"020dc3fe-e94c-444d-b968-021fcd129218\") "
Jul 16 22:06:11.338583 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.338548    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020dc3fe-e94c-444d-b968-021fcd129218-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "020dc3fe-e94c-444d-b968-021fcd129218" (UID: "020dc3fe-e94c-444d-b968-021fcd129218"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:06:11.338583 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.338570    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020dc3fe-e94c-444d-b968-021fcd129218-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "020dc3fe-e94c-444d-b968-021fcd129218" (UID: "020dc3fe-e94c-444d-b968-021fcd129218"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:06:11.338718 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.338696    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020dc3fe-e94c-444d-b968-021fcd129218-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "020dc3fe-e94c-444d-b968-021fcd129218" (UID: "020dc3fe-e94c-444d-b968-021fcd129218"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:06:11.340300 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.340281    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020dc3fe-e94c-444d-b968-021fcd129218-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "020dc3fe-e94c-444d-b968-021fcd129218" (UID: "020dc3fe-e94c-444d-b968-021fcd129218"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:06:11.340368 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.340337    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020dc3fe-e94c-444d-b968-021fcd129218-kube-api-access-pv44n" (OuterVolumeSpecName: "kube-api-access-pv44n") pod "020dc3fe-e94c-444d-b968-021fcd129218" (UID: "020dc3fe-e94c-444d-b968-021fcd129218"). InnerVolumeSpecName "kube-api-access-pv44n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:06:11.385896 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.385865    2570 generic.go:356] "Generic (PLEG): container finished" podID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerID="60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed" exitCode=0
Jul 16 22:06:11.386238 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.385930    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"
Jul 16 22:06:11.386238 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.385951    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" event={"ID":"e9b499bf-dfe6-451b-b5c4-81aeb539eab7","Type":"ContainerDied","Data":"60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed"}
Jul 16 22:06:11.386238 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.386014    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj" event={"ID":"e9b499bf-dfe6-451b-b5c4-81aeb539eab7","Type":"ContainerDied","Data":"dbb87caa55d0bf8ee447889840523307c6607fab52b67b1a30181348f2c1f376"}
Jul 16 22:06:11.386238 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.386050    2570 scope.go:117] "RemoveContainer" containerID="bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7"
Jul 16 22:06:11.388010 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.387984    2570 generic.go:356] "Generic (PLEG): container finished" podID="020dc3fe-e94c-444d-b968-021fcd129218" containerID="a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01" exitCode=0
Jul 16 22:06:11.388128 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.388039    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"
Jul 16 22:06:11.388128 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.388050    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" event={"ID":"020dc3fe-e94c-444d-b968-021fcd129218","Type":"ContainerDied","Data":"a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01"}
Jul 16 22:06:11.388128 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.388121    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd" event={"ID":"020dc3fe-e94c-444d-b968-021fcd129218","Type":"ContainerDied","Data":"8bf2406edeea362ae57f1bbd0bdb0de7bb9525a0de73ecd01e8dec7fbd1044f1"}
Jul 16 22:06:11.394666 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.394451    2570 scope.go:117] "RemoveContainer" containerID="60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed"
Jul 16 22:06:11.401606 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.401584    2570 scope.go:117] "RemoveContainer" containerID="2976ea7b6361e8e4f4a9ced4c56a6a96dc4565e010139c1fc91cdf780a2b694f"
Jul 16 22:06:11.408030 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.408013    2570 scope.go:117] "RemoveContainer" containerID="bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7"
Jul 16 22:06:11.408429 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:06:11.408388    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7\": container with ID starting with bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7 not found: ID does not exist" containerID="bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7"
Jul 16 22:06:11.408538 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.408425    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7"} err="failed to get container status \"bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7\": rpc error: code = NotFound desc = could not find container \"bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7\": container with ID starting with bb1b7c11efb422499a99def824802988e6dedc785db0ca4e7d8e35cdcfe955f7 not found: ID does not exist"
Jul 16 22:06:11.408538 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.408448    2570 scope.go:117] "RemoveContainer" containerID="60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed"
Jul 16 22:06:11.408962 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:06:11.408930    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed\": container with ID starting with 60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed not found: ID does not exist" containerID="60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed"
Jul 16 22:06:11.408962 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.408954    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed"} err="failed to get container status \"60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed\": rpc error: code = NotFound desc = could not find container \"60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed\": container with ID starting with 60e4e865925025a624e0cce675c9d732647a2dc0d7b56cc7ddecaf0a450982ed not found: ID does not exist"
Jul 16 22:06:11.409128 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.408971    2570 scope.go:117] "RemoveContainer" containerID="2976ea7b6361e8e4f4a9ced4c56a6a96dc4565e010139c1fc91cdf780a2b694f"
Jul 16 22:06:11.409263 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:06:11.409241    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2976ea7b6361e8e4f4a9ced4c56a6a96dc4565e010139c1fc91cdf780a2b694f\": container with ID starting with 2976ea7b6361e8e4f4a9ced4c56a6a96dc4565e010139c1fc91cdf780a2b694f not found: ID does not exist" containerID="2976ea7b6361e8e4f4a9ced4c56a6a96dc4565e010139c1fc91cdf780a2b694f"
Jul 16 22:06:11.409320 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.409271    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2976ea7b6361e8e4f4a9ced4c56a6a96dc4565e010139c1fc91cdf780a2b694f"} err="failed to get container status \"2976ea7b6361e8e4f4a9ced4c56a6a96dc4565e010139c1fc91cdf780a2b694f\": rpc error: code = NotFound desc = could not find container \"2976ea7b6361e8e4f4a9ced4c56a6a96dc4565e010139c1fc91cdf780a2b694f\": container with ID starting with 2976ea7b6361e8e4f4a9ced4c56a6a96dc4565e010139c1fc91cdf780a2b694f not found: ID does not exist"
Jul 16 22:06:11.409320 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.409296    2570 scope.go:117] "RemoveContainer" containerID="343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72"
Jul 16 22:06:11.410590 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.410571    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"]
Jul 16 22:06:11.414370 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.414348    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj"]
Jul 16 22:06:11.416453 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.416437    2570 scope.go:117] "RemoveContainer" containerID="a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01"
Jul 16 22:06:11.423044 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.423021    2570 scope.go:117] "RemoveContainer" containerID="716788414c4265896561b093108a47d0cd1f7bd18a75b33308e5e61d791c5c49"
Jul 16 22:06:11.424426 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.424409    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"]
Jul 16 22:06:11.427692 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.427670    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd"]
Jul 16 22:06:11.429852 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.429837    2570 scope.go:117] "RemoveContainer" containerID="343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72"
Jul 16 22:06:11.430141 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:06:11.430121    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72\": container with ID starting with 343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72 not found: ID does not exist" containerID="343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72"
Jul 16 22:06:11.430209 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.430150    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72"} err="failed to get container status \"343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72\": rpc error: code = NotFound desc = could not find container \"343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72\": container with ID starting with 343d3e25c74c8fde8f2897f0cfb0ac6fd029c6992745b96f21c1c0f6d1e35b72 not found: ID does not exist"
Jul 16 22:06:11.430209 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.430175    2570 scope.go:117] "RemoveContainer" containerID="a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01"
Jul 16 22:06:11.430435 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:06:11.430419    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01\": container with ID starting with a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01 not found: ID does not exist" containerID="a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01"
Jul 16 22:06:11.430476 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.430439    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01"} err="failed to get container status \"a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01\": rpc error: code = NotFound desc = could not find container \"a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01\": container with ID starting with a00e5f21bddc6a951f3f164a92f5eb2546f0d56d57ded34335e80b3095178a01 not found: ID does not exist"
Jul 16 22:06:11.430476 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.430454    2570 scope.go:117] "RemoveContainer" containerID="716788414c4265896561b093108a47d0cd1f7bd18a75b33308e5e61d791c5c49"
Jul 16 22:06:11.430668 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:06:11.430652    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716788414c4265896561b093108a47d0cd1f7bd18a75b33308e5e61d791c5c49\": container with ID starting with 716788414c4265896561b093108a47d0cd1f7bd18a75b33308e5e61d791c5c49 not found: ID does not exist" containerID="716788414c4265896561b093108a47d0cd1f7bd18a75b33308e5e61d791c5c49"
Jul 16 22:06:11.430711 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.430674    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716788414c4265896561b093108a47d0cd1f7bd18a75b33308e5e61d791c5c49"} err="failed to get container status \"716788414c4265896561b093108a47d0cd1f7bd18a75b33308e5e61d791c5c49\": rpc error: code = NotFound desc = could not find container \"716788414c4265896561b093108a47d0cd1f7bd18a75b33308e5e61d791c5c49\": container with ID starting with 716788414c4265896561b093108a47d0cd1f7bd18a75b33308e5e61d791c5c49 not found: ID does not exist"
Jul 16 22:06:11.439035 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.439014    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020dc3fe-e94c-444d-b968-021fcd129218-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:06:11.439150 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.439038    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv44n\" (UniqueName: \"kubernetes.io/projected/020dc3fe-e94c-444d-b968-021fcd129218-kube-api-access-pv44n\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:06:11.439150 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.439052    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020dc3fe-e94c-444d-b968-021fcd129218-cabundle-cert\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:06:11.439150 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.439067    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020dc3fe-e94c-444d-b968-021fcd129218-kserve-provision-location\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:06:11.439150 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.439104    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/020dc3fe-e94c-444d-b968-021fcd129218-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:06:11.871741 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.871712    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="020dc3fe-e94c-444d-b968-021fcd129218" path="/var/lib/kubelet/pods/020dc3fe-e94c-444d-b968-021fcd129218/volumes"
Jul 16 22:06:11.872242 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:11.872225    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" path="/var/lib/kubelet/pods/e9b499bf-dfe6-451b-b5c4-81aeb539eab7/volumes"
Jul 16 22:06:17.398014 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:17.397982    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:06:26.372876 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.372836    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"]
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373118    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="storage-initializer"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373131    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="storage-initializer"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373140    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kube-rbac-proxy"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373145    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kube-rbac-proxy"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373151    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kserve-container"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373157    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kserve-container"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373164    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kube-rbac-proxy"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373169    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kube-rbac-proxy"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373181    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="storage-initializer"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373186    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="storage-initializer"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373198    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373203    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373248    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kube-rbac-proxy"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373256    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kube-rbac-proxy"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373261    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9b499bf-dfe6-451b-b5c4-81aeb539eab7" containerName="kserve-container"
Jul 16 22:06:26.373296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.373266    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="020dc3fe-e94c-444d-b968-021fcd129218" containerName="kserve-container"
Jul 16 22:06:26.377568 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.377551    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.382774 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.382747    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:06:26.382908 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.382821    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-164a6-kube-rbac-proxy-sar-config\""
Jul 16 22:06:26.383075 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.383056    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-164a6-predictor-serving-cert\""
Jul 16 22:06:26.383225 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.383103    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 22:06:26.399348 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.399324    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"]
Jul 16 22:06:26.439518 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.439491    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/419f3623-53f4-46ef-bfea-7f159c51321d-cabundle-cert\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.439658 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.439536    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419f3623-53f4-46ef-bfea-7f159c51321d-proxy-tls\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.439658 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.439592    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419f3623-53f4-46ef-bfea-7f159c51321d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.439658 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.439621    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzhmm\" (UniqueName: \"kubernetes.io/projected/419f3623-53f4-46ef-bfea-7f159c51321d-kube-api-access-xzhmm\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.439824 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.439711    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/419f3623-53f4-46ef-bfea-7f159c51321d-kserve-provision-location\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.540561 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.540521    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/419f3623-53f4-46ef-bfea-7f159c51321d-cabundle-cert\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.540799 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.540572    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419f3623-53f4-46ef-bfea-7f159c51321d-proxy-tls\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.540799 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.540614    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419f3623-53f4-46ef-bfea-7f159c51321d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.540799 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.540651    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzhmm\" (UniqueName: \"kubernetes.io/projected/419f3623-53f4-46ef-bfea-7f159c51321d-kube-api-access-xzhmm\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.540799 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.540694    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/419f3623-53f4-46ef-bfea-7f159c51321d-kserve-provision-location\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.540799 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:06:26.540761    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-serving-cert: secret "isvc-logger-raw-164a6-predictor-serving-cert" not found
Jul 16 22:06:26.541065 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:06:26.540834    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419f3623-53f4-46ef-bfea-7f159c51321d-proxy-tls podName:419f3623-53f4-46ef-bfea-7f159c51321d nodeName:}" failed. No retries permitted until 2026-07-16 22:06:27.04081265 +0000 UTC m=+971.638352867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/419f3623-53f4-46ef-bfea-7f159c51321d-proxy-tls") pod "isvc-logger-raw-164a6-predictor-747b66f544-qkntm" (UID: "419f3623-53f4-46ef-bfea-7f159c51321d") : secret "isvc-logger-raw-164a6-predictor-serving-cert" not found
Jul 16 22:06:26.541147 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.541109    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/419f3623-53f4-46ef-bfea-7f159c51321d-kserve-provision-location\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.541267 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.541247    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/419f3623-53f4-46ef-bfea-7f159c51321d-cabundle-cert\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.541308 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.541291    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419f3623-53f4-46ef-bfea-7f159c51321d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:26.561575 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:26.561554    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzhmm\" (UniqueName: \"kubernetes.io/projected/419f3623-53f4-46ef-bfea-7f159c51321d-kube-api-access-xzhmm\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:27.045131 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:27.045105    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419f3623-53f4-46ef-bfea-7f159c51321d-proxy-tls\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:27.047406 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:27.047382    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419f3623-53f4-46ef-bfea-7f159c51321d-proxy-tls\") pod \"isvc-logger-raw-164a6-predictor-747b66f544-qkntm\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:27.287279 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:27.287249    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:27.424553 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:27.424522    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"]
Jul 16 22:06:27.428133 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:06:27.428095    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419f3623_53f4_46ef_bfea_7f159c51321d.slice/crio-41b84cbd0b28f6a0d9787fa019236f734d893e7e47d6e2f2d0c33b10afc586f6 WatchSource:0}: Error finding container 41b84cbd0b28f6a0d9787fa019236f734d893e7e47d6e2f2d0c33b10afc586f6: Status 404 returned error can't find the container with id 41b84cbd0b28f6a0d9787fa019236f734d893e7e47d6e2f2d0c33b10afc586f6
Jul 16 22:06:27.449922 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:27.449891    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" event={"ID":"419f3623-53f4-46ef-bfea-7f159c51321d","Type":"ContainerStarted","Data":"41b84cbd0b28f6a0d9787fa019236f734d893e7e47d6e2f2d0c33b10afc586f6"}
Jul 16 22:06:28.455487 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:28.455448    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" event={"ID":"419f3623-53f4-46ef-bfea-7f159c51321d","Type":"ContainerStarted","Data":"af5eaf6bf7d7b0de6969ea64e572d0bd433b2c16648ca8f2b6dbaacb5693b924"}
Jul 16 22:06:31.467787 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:31.467693    2570 generic.go:356] "Generic (PLEG): container finished" podID="419f3623-53f4-46ef-bfea-7f159c51321d" containerID="af5eaf6bf7d7b0de6969ea64e572d0bd433b2c16648ca8f2b6dbaacb5693b924" exitCode=0
Jul 16 22:06:31.467787 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:31.467766    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" event={"ID":"419f3623-53f4-46ef-bfea-7f159c51321d","Type":"ContainerDied","Data":"af5eaf6bf7d7b0de6969ea64e572d0bd433b2c16648ca8f2b6dbaacb5693b924"}
Jul 16 22:06:32.474162 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:32.474122    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" event={"ID":"419f3623-53f4-46ef-bfea-7f159c51321d","Type":"ContainerStarted","Data":"f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242"}
Jul 16 22:06:32.474729 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:32.474207    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" event={"ID":"419f3623-53f4-46ef-bfea-7f159c51321d","Type":"ContainerStarted","Data":"feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011"}
Jul 16 22:06:32.474729 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:32.474254    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" event={"ID":"419f3623-53f4-46ef-bfea-7f159c51321d","Type":"ContainerStarted","Data":"181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161"}
Jul 16 22:06:32.474729 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:32.474505    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:32.500236 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:32.500189    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podStartSLOduration=2.785942086 podStartE2EDuration="6.50017465s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.714232564s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:06:32.498225701 +0000 UTC m=+977.095765925" watchObservedRunningTime="2026-07-16 22:06:32.50017465 +0000 UTC m=+977.097714880"
Jul 16 22:06:33.477651 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:33.477618    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:33.478011 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:33.477695    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:33.479119 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:33.479072    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 16 22:06:33.479666 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:33.479640    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:06:34.481306 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:34.481262    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 16 22:06:34.481780 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:34.481717    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:06:39.485145 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:39.485061    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:06:39.485680 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:39.485651    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 16 22:06:39.485948 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:39.485922    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:06:49.486334 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:49.486289    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 16 22:06:49.486805 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:49.486779    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:06:59.486517 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:59.486469    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 16 22:06:59.486936 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:06:59.486908    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:07:09.485601 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:07:09.485548    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 16 22:07:09.486201 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:07:09.486130    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:07:19.486131 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:07:19.486062    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 16 22:07:19.486540 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:07:19.486480    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:07:29.486169 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:07:29.486127    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 16 22:07:29.486562 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:07:29.486500    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:07:39.486299 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:07:39.486250    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 16 22:07:39.486779 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:07:39.486754    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:07:49.486133 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:07:49.486076    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:07:49.486603 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:07:49.486573    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:08:01.374784 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.374757    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr_5d3de467-678e-4ac6-a6ea-55c2192dcae4/kserve-container/0.log"
Jul 16 22:08:01.596688 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.596651    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"]
Jul 16 22:08:01.596998 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.596968    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" containerID="cri-o://181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161" gracePeriod=30
Jul 16 22:08:01.597053 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.597009    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kube-rbac-proxy" containerID="cri-o://feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011" gracePeriod=30
Jul 16 22:08:01.597179 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.596977    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" containerID="cri-o://f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242" gracePeriod=30
Jul 16 22:08:01.712265 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.712234    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"]
Jul 16 22:08:01.712503 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.712473    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr" podUID="5d3de467-678e-4ac6-a6ea-55c2192dcae4" containerName="kserve-container" containerID="cri-o://49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174" gracePeriod=30
Jul 16 22:08:01.712624 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.712520    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr" podUID="5d3de467-678e-4ac6-a6ea-55c2192dcae4" containerName="kube-rbac-proxy" containerID="cri-o://ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5" gracePeriod=30
Jul 16 22:08:01.790238 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.790209    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"]
Jul 16 22:08:01.793524 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.793505    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.795851 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.795833    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-e4053-predictor-serving-cert\""
Jul 16 22:08:01.795966 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.795863    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-e4053-kube-rbac-proxy-sar-config\""
Jul 16 22:08:01.801820 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.801796    2570 generic.go:356] "Generic (PLEG): container finished" podID="419f3623-53f4-46ef-bfea-7f159c51321d" containerID="feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011" exitCode=2
Jul 16 22:08:01.801911 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.801830    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" event={"ID":"419f3623-53f4-46ef-bfea-7f159c51321d","Type":"ContainerDied","Data":"feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011"}
Jul 16 22:08:01.805319 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.805300    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"]
Jul 16 22:08:01.834895 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.834870    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.835015 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.834906    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjkz\" (UniqueName: \"kubernetes.io/projected/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kube-api-access-9wjkz\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.835015 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.834923    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-cabundle-cert\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.835127 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.835016    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-proxy-tls\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.835127 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.835033    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.936080 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.936049    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-proxy-tls\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.936080 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.936102    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.936332 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.936186    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.936332 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.936217    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjkz\" (UniqueName: \"kubernetes.io/projected/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kube-api-access-9wjkz\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.936332 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.936240    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-cabundle-cert\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.936616 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.936593    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.936978 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.936954    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-cabundle-cert\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.937062 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.937014    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.938737 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.938713    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-proxy-tls\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.944815 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.944794    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjkz\" (UniqueName: \"kubernetes.io/projected/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kube-api-access-9wjkz\") pod \"isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:01.958658 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:01.958638    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:08:02.036668 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.036638    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d3de467-678e-4ac6-a6ea-55c2192dcae4-proxy-tls\") pod \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") "
Jul 16 22:08:02.036668 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.036673    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjmnt\" (UniqueName: \"kubernetes.io/projected/5d3de467-678e-4ac6-a6ea-55c2192dcae4-kube-api-access-mjmnt\") pod \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") "
Jul 16 22:08:02.036878 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.036707    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d3de467-678e-4ac6-a6ea-55c2192dcae4-kube-rbac-proxy-sar-config\") pod \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\" (UID: \"5d3de467-678e-4ac6-a6ea-55c2192dcae4\") "
Jul 16 22:08:02.037104 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.037056    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3de467-678e-4ac6-a6ea-55c2192dcae4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5d3de467-678e-4ac6-a6ea-55c2192dcae4" (UID: "5d3de467-678e-4ac6-a6ea-55c2192dcae4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:08:02.038727 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.038707    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3de467-678e-4ac6-a6ea-55c2192dcae4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d3de467-678e-4ac6-a6ea-55c2192dcae4" (UID: "5d3de467-678e-4ac6-a6ea-55c2192dcae4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:08:02.038791 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.038746    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3de467-678e-4ac6-a6ea-55c2192dcae4-kube-api-access-mjmnt" (OuterVolumeSpecName: "kube-api-access-mjmnt") pod "5d3de467-678e-4ac6-a6ea-55c2192dcae4" (UID: "5d3de467-678e-4ac6-a6ea-55c2192dcae4"). InnerVolumeSpecName "kube-api-access-mjmnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:08:02.105181 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.105143    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:02.138204 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.138168    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d3de467-678e-4ac6-a6ea-55c2192dcae4-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:08:02.138204 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.138203    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjmnt\" (UniqueName: \"kubernetes.io/projected/5d3de467-678e-4ac6-a6ea-55c2192dcae4-kube-api-access-mjmnt\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:08:02.138377 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.138215    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d3de467-678e-4ac6-a6ea-55c2192dcae4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:08:02.234555 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.234522    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"]
Jul 16 22:08:02.237573 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:08:02.237548    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4d1fa8_49c5_454f_9641_c4baaf90fe78.slice/crio-3f3848f7ad5fb3ea3ad6d40d1e1167e130dc8487ff8b15e73c1874a072f12621 WatchSource:0}: Error finding container 3f3848f7ad5fb3ea3ad6d40d1e1167e130dc8487ff8b15e73c1874a072f12621: Status 404 returned error can't find the container with id 3f3848f7ad5fb3ea3ad6d40d1e1167e130dc8487ff8b15e73c1874a072f12621
Jul 16 22:08:02.807357 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.807319    2570 generic.go:356] "Generic (PLEG): container finished" podID="5d3de467-678e-4ac6-a6ea-55c2192dcae4" containerID="ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5" exitCode=2
Jul 16 22:08:02.807357 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.807350    2570 generic.go:356] "Generic (PLEG): container finished" podID="5d3de467-678e-4ac6-a6ea-55c2192dcae4" containerID="49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174" exitCode=2
Jul 16 22:08:02.807839 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.807376    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"
Jul 16 22:08:02.807839 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.807396    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr" event={"ID":"5d3de467-678e-4ac6-a6ea-55c2192dcae4","Type":"ContainerDied","Data":"ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5"}
Jul 16 22:08:02.807839 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.807450    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr" event={"ID":"5d3de467-678e-4ac6-a6ea-55c2192dcae4","Type":"ContainerDied","Data":"49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174"}
Jul 16 22:08:02.807839 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.807477    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr" event={"ID":"5d3de467-678e-4ac6-a6ea-55c2192dcae4","Type":"ContainerDied","Data":"ac9e91e454c2d23ea48e56c1239ba59685465b2e619b22a1cb77b14cac184532"}
Jul 16 22:08:02.807839 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.807503    2570 scope.go:117] "RemoveContainer" containerID="ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5"
Jul 16 22:08:02.809052 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.809026    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" event={"ID":"8f4d1fa8-49c5-454f-9641-c4baaf90fe78","Type":"ContainerStarted","Data":"693bc7778d868117174f840161e7a5969f2888281c018b4636088bf966bf7ad7"}
Jul 16 22:08:02.809174 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.809116    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" event={"ID":"8f4d1fa8-49c5-454f-9641-c4baaf90fe78","Type":"ContainerStarted","Data":"3f3848f7ad5fb3ea3ad6d40d1e1167e130dc8487ff8b15e73c1874a072f12621"}
Jul 16 22:08:02.817206 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.817188    2570 scope.go:117] "RemoveContainer" containerID="49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174"
Jul 16 22:08:02.825754 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.825736    2570 scope.go:117] "RemoveContainer" containerID="ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5"
Jul 16 22:08:02.826045 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:08:02.826019    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5\": container with ID starting with ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5 not found: ID does not exist" containerID="ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5"
Jul 16 22:08:02.826137 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.826053    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5"} err="failed to get container status \"ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5\": rpc error: code = NotFound desc = could not find container \"ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5\": container with ID starting with ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5 not found: ID does not exist"
Jul 16 22:08:02.826137 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.826073    2570 scope.go:117] "RemoveContainer" containerID="49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174"
Jul 16 22:08:02.826405 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:08:02.826386    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174\": container with ID starting with 49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174 not found: ID does not exist" containerID="49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174"
Jul 16 22:08:02.826459 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.826410    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174"} err="failed to get container status \"49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174\": rpc error: code = NotFound desc = could not find container \"49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174\": container with ID starting with 49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174 not found: ID does not exist"
Jul 16 22:08:02.826459 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.826427    2570 scope.go:117] "RemoveContainer" containerID="ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5"
Jul 16 22:08:02.826716 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.826693    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5"} err="failed to get container status \"ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5\": rpc error: code = NotFound desc = could not find container \"ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5\": container with ID starting with ace575de8e60624d3807fe8d529190c93465c9c5e6f4b3e686122d08151fbda5 not found: ID does not exist"
Jul 16 22:08:02.826770 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.826718    2570 scope.go:117] "RemoveContainer" containerID="49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174"
Jul 16 22:08:02.831401 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.827257    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174"} err="failed to get container status \"49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174\": rpc error: code = NotFound desc = could not find container \"49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174\": container with ID starting with 49e4017b5bc486a6e9e5b1e66b877416c1c9b69dac61ef69f4d19a7acc8b1174 not found: ID does not exist"
Jul 16 22:08:02.849017 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.848992    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"]
Jul 16 22:08:02.854686 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:02.854663    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr"]
Jul 16 22:08:03.873076 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:03.873042    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d3de467-678e-4ac6-a6ea-55c2192dcae4" path="/var/lib/kubelet/pods/5d3de467-678e-4ac6-a6ea-55c2192dcae4/volumes"
Jul 16 22:08:04.481775 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:04.481731    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 16 22:08:06.827364 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:06.827332    2570 generic.go:356] "Generic (PLEG): container finished" podID="419f3623-53f4-46ef-bfea-7f159c51321d" containerID="181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161" exitCode=0
Jul 16 22:08:06.827754 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:06.827406    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" event={"ID":"419f3623-53f4-46ef-bfea-7f159c51321d","Type":"ContainerDied","Data":"181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161"}
Jul 16 22:08:06.828659 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:06.828640    2570 generic.go:356] "Generic (PLEG): container finished" podID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerID="693bc7778d868117174f840161e7a5969f2888281c018b4636088bf966bf7ad7" exitCode=0
Jul 16 22:08:06.828737 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:06.828716    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" event={"ID":"8f4d1fa8-49c5-454f-9641-c4baaf90fe78","Type":"ContainerDied","Data":"693bc7778d868117174f840161e7a5969f2888281c018b4636088bf966bf7ad7"}
Jul 16 22:08:07.834873 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:07.834841    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" event={"ID":"8f4d1fa8-49c5-454f-9641-c4baaf90fe78","Type":"ContainerStarted","Data":"65714bac0cd1e5ac1897c55e7dc0b7bff425ddc8cf365b43627da78ffaa7ac29"}
Jul 16 22:08:07.835357 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:07.834942    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" event={"ID":"8f4d1fa8-49c5-454f-9641-c4baaf90fe78","Type":"ContainerStarted","Data":"0c554e334ecd1763296ec66e63608db491efa196c55bcb4a1bfdcfeaa79c0b41"}
Jul 16 22:08:07.835357 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:07.835169    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:07.860654 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:07.860605    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podStartSLOduration=3.161688673 podStartE2EDuration="6.860591765s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698903092s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:08:07.858869493 +0000 UTC m=+1072.456409726" watchObservedRunningTime="2026-07-16 22:08:07.860591765 +0000 UTC m=+1072.458131999"
Jul 16 22:08:08.838834 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:08.838802    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:08.840114 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:08.840068    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:08:09.482196 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:09.482153    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 16 22:08:09.486778 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:09.486746    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 16 22:08:09.486936 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:09.486829    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:08:09.842630 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:09.842540    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:08:14.481919 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:14.481871    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 16 22:08:14.482296 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:14.482012    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:08:14.846831 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:14.846754    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:08:14.847265 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:14.847240    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:08:19.481719 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:19.481673    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 16 22:08:19.486378 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:19.486348    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:08:19.486613 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:19.486590    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 16 22:08:24.481981 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:24.481935    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 16 22:08:24.847995 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:24.847907    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:08:29.482405 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:29.482355    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 16 22:08:29.486150 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:29.486111    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:08:29.486257 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:29.486244    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:08:29.486616 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:29.486595    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 16 22:08:29.486715 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:29.486703    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:08:31.745998 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.745975    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:08:31.862960 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.862894    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/419f3623-53f4-46ef-bfea-7f159c51321d-kserve-provision-location\") pod \"419f3623-53f4-46ef-bfea-7f159c51321d\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") "
Jul 16 22:08:31.862960 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.862935    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/419f3623-53f4-46ef-bfea-7f159c51321d-cabundle-cert\") pod \"419f3623-53f4-46ef-bfea-7f159c51321d\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") "
Jul 16 22:08:31.862960 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.862951    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419f3623-53f4-46ef-bfea-7f159c51321d-kube-rbac-proxy-sar-config\") pod \"419f3623-53f4-46ef-bfea-7f159c51321d\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") "
Jul 16 22:08:31.863193 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.862977    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzhmm\" (UniqueName: \"kubernetes.io/projected/419f3623-53f4-46ef-bfea-7f159c51321d-kube-api-access-xzhmm\") pod \"419f3623-53f4-46ef-bfea-7f159c51321d\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") "
Jul 16 22:08:31.863193 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.863028    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419f3623-53f4-46ef-bfea-7f159c51321d-proxy-tls\") pod \"419f3623-53f4-46ef-bfea-7f159c51321d\" (UID: \"419f3623-53f4-46ef-bfea-7f159c51321d\") "
Jul 16 22:08:31.863276 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.863251    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419f3623-53f4-46ef-bfea-7f159c51321d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "419f3623-53f4-46ef-bfea-7f159c51321d" (UID: "419f3623-53f4-46ef-bfea-7f159c51321d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:08:31.863334 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.863313    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419f3623-53f4-46ef-bfea-7f159c51321d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "419f3623-53f4-46ef-bfea-7f159c51321d" (UID: "419f3623-53f4-46ef-bfea-7f159c51321d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:08:31.863377 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.863311    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419f3623-53f4-46ef-bfea-7f159c51321d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "419f3623-53f4-46ef-bfea-7f159c51321d" (UID: "419f3623-53f4-46ef-bfea-7f159c51321d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:08:31.865118 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.865070    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419f3623-53f4-46ef-bfea-7f159c51321d-kube-api-access-xzhmm" (OuterVolumeSpecName: "kube-api-access-xzhmm") pod "419f3623-53f4-46ef-bfea-7f159c51321d" (UID: "419f3623-53f4-46ef-bfea-7f159c51321d"). InnerVolumeSpecName "kube-api-access-xzhmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:08:31.865118 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.865075    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/419f3623-53f4-46ef-bfea-7f159c51321d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "419f3623-53f4-46ef-bfea-7f159c51321d" (UID: "419f3623-53f4-46ef-bfea-7f159c51321d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:08:31.919246 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.919222    2570 generic.go:356] "Generic (PLEG): container finished" podID="419f3623-53f4-46ef-bfea-7f159c51321d" containerID="f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242" exitCode=0
Jul 16 22:08:31.919330 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.919273    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" event={"ID":"419f3623-53f4-46ef-bfea-7f159c51321d","Type":"ContainerDied","Data":"f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242"}
Jul 16 22:08:31.919330 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.919288    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"
Jul 16 22:08:31.919330 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.919324    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm" event={"ID":"419f3623-53f4-46ef-bfea-7f159c51321d","Type":"ContainerDied","Data":"41b84cbd0b28f6a0d9787fa019236f734d893e7e47d6e2f2d0c33b10afc586f6"}
Jul 16 22:08:31.919426 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.919356    2570 scope.go:117] "RemoveContainer" containerID="f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242"
Jul 16 22:08:31.927038 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.927024    2570 scope.go:117] "RemoveContainer" containerID="feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011"
Jul 16 22:08:31.933905 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.933890    2570 scope.go:117] "RemoveContainer" containerID="181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161"
Jul 16 22:08:31.936321 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.936300    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"]
Jul 16 22:08:31.940692 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.940669    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm"]
Jul 16 22:08:31.941624 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.941599    2570 scope.go:117] "RemoveContainer" containerID="af5eaf6bf7d7b0de6969ea64e572d0bd433b2c16648ca8f2b6dbaacb5693b924"
Jul 16 22:08:31.948058 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.948043    2570 scope.go:117] "RemoveContainer" containerID="f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242"
Jul 16 22:08:31.948313 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:08:31.948294    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242\": container with ID starting with f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242 not found: ID does not exist" containerID="f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242"
Jul 16 22:08:31.948379 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.948319    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242"} err="failed to get container status \"f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242\": rpc error: code = NotFound desc = could not find container \"f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242\": container with ID starting with f40b7a786c19427d9bf0ac1fa48cc675b71e7b8ff42cc768f52dd85fb8192242 not found: ID does not exist"
Jul 16 22:08:31.948379 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.948336    2570 scope.go:117] "RemoveContainer" containerID="feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011"
Jul 16 22:08:31.948576 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:08:31.948558    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011\": container with ID starting with feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011 not found: ID does not exist" containerID="feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011"
Jul 16 22:08:31.948662 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.948581    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011"} err="failed to get container status \"feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011\": rpc error: code = NotFound desc = could not find container \"feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011\": container with ID starting with feb48eae61e46e778fc74abb0f86707936152c01bbceb5ef11dc8b050494f011 not found: ID does not exist"
Jul 16 22:08:31.948662 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.948594    2570 scope.go:117] "RemoveContainer" containerID="181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161"
Jul 16 22:08:31.948814 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:08:31.948799    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161\": container with ID starting with 181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161 not found: ID does not exist" containerID="181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161"
Jul 16 22:08:31.948855 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.948818    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161"} err="failed to get container status \"181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161\": rpc error: code = NotFound desc = could not find container \"181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161\": container with ID starting with 181ef84a1c4c4530c85ed7966cfa8a798179777775fb2cf8563e6af48f2ca161 not found: ID does not exist"
Jul 16 22:08:31.948855 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.948830    2570 scope.go:117] "RemoveContainer" containerID="af5eaf6bf7d7b0de6969ea64e572d0bd433b2c16648ca8f2b6dbaacb5693b924"
Jul 16 22:08:31.949021 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:08:31.949006    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5eaf6bf7d7b0de6969ea64e572d0bd433b2c16648ca8f2b6dbaacb5693b924\": container with ID starting with af5eaf6bf7d7b0de6969ea64e572d0bd433b2c16648ca8f2b6dbaacb5693b924 not found: ID does not exist" containerID="af5eaf6bf7d7b0de6969ea64e572d0bd433b2c16648ca8f2b6dbaacb5693b924"
Jul 16 22:08:31.949063 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.949023    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5eaf6bf7d7b0de6969ea64e572d0bd433b2c16648ca8f2b6dbaacb5693b924"} err="failed to get container status \"af5eaf6bf7d7b0de6969ea64e572d0bd433b2c16648ca8f2b6dbaacb5693b924\": rpc error: code = NotFound desc = could not find container \"af5eaf6bf7d7b0de6969ea64e572d0bd433b2c16648ca8f2b6dbaacb5693b924\": container with ID starting with af5eaf6bf7d7b0de6969ea64e572d0bd433b2c16648ca8f2b6dbaacb5693b924 not found: ID does not exist"
Jul 16 22:08:31.964347 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.964324    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/419f3623-53f4-46ef-bfea-7f159c51321d-cabundle-cert\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:08:31.964347 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.964345    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419f3623-53f4-46ef-bfea-7f159c51321d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:08:31.964448 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.964354    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzhmm\" (UniqueName: \"kubernetes.io/projected/419f3623-53f4-46ef-bfea-7f159c51321d-kube-api-access-xzhmm\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:08:31.964448 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.964363    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419f3623-53f4-46ef-bfea-7f159c51321d-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:08:31.964448 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:31.964374    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/419f3623-53f4-46ef-bfea-7f159c51321d-kserve-provision-location\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:08:33.871261 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:33.871234    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" path="/var/lib/kubelet/pods/419f3623-53f4-46ef-bfea-7f159c51321d/volumes"
Jul 16 22:08:34.848025 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:34.847990    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:08:44.847217 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:44.847172    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:08:54.847891 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:08:54.847854    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:09:04.847853 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:09:04.847811    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:09:14.848125 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:09:14.848059    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:09:22.866702 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:09:22.866656    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:09:32.866688 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:09:32.866639    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:09:42.866667 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:09:42.866558    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:09:52.867118 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:09:52.867058    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:10:02.866939 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:02.866892    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:10:12.866827 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:12.866783    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:10:15.832922 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:15.832893    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 22:10:15.834388 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:15.834362    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 22:10:22.867298 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:22.867248    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:10:32.867236 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:32.867204    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:10:41.851070 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:41.851033    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"]
Jul 16 22:10:41.851507 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:41.851302    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" containerID="cri-o://0c554e334ecd1763296ec66e63608db491efa196c55bcb4a1bfdcfeaa79c0b41" gracePeriod=30
Jul 16 22:10:41.851507 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:41.851322    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kube-rbac-proxy" containerID="cri-o://65714bac0cd1e5ac1897c55e7dc0b7bff425ddc8cf365b43627da78ffaa7ac29" gracePeriod=30
Jul 16 22:10:42.370014 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:42.369976    2570 generic.go:356] "Generic (PLEG): container finished" podID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerID="65714bac0cd1e5ac1897c55e7dc0b7bff425ddc8cf365b43627da78ffaa7ac29" exitCode=2
Jul 16 22:10:42.370206 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:42.370048    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" event={"ID":"8f4d1fa8-49c5-454f-9641-c4baaf90fe78","Type":"ContainerDied","Data":"65714bac0cd1e5ac1897c55e7dc0b7bff425ddc8cf365b43627da78ffaa7ac29"}
Jul 16 22:10:42.866892 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:42.866851    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 22:10:44.842845 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:44.842808    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 16 22:10:47.053562 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.053528    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"]
Jul 16 22:10:47.053938 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.053899    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="storage-initializer"
Jul 16 22:10:47.053938 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.053913    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="storage-initializer"
Jul 16 22:10:47.053938 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.053928    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kube-rbac-proxy"
Jul 16 22:10:47.053938 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.053937    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kube-rbac-proxy"
Jul 16 22:10:47.054070 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.053955    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent"
Jul 16 22:10:47.054070 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.053964    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent"
Jul 16 22:10:47.054070 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.053978    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d3de467-678e-4ac6-a6ea-55c2192dcae4" containerName="kserve-container"
Jul 16 22:10:47.054070 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.053986    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d3de467-678e-4ac6-a6ea-55c2192dcae4" containerName="kserve-container"
Jul 16 22:10:47.054070 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.054001    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d3de467-678e-4ac6-a6ea-55c2192dcae4" containerName="kube-rbac-proxy"
Jul 16 22:10:47.054070 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.054009    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d3de467-678e-4ac6-a6ea-55c2192dcae4" containerName="kube-rbac-proxy"
Jul 16 22:10:47.054070 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.054022    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container"
Jul 16 22:10:47.054070 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.054030    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container"
Jul 16 22:10:47.054473 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.054107    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d3de467-678e-4ac6-a6ea-55c2192dcae4" containerName="kube-rbac-proxy"
Jul 16 22:10:47.054473 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.054119    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d3de467-678e-4ac6-a6ea-55c2192dcae4" containerName="kserve-container"
Jul 16 22:10:47.054473 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.054129    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kube-rbac-proxy"
Jul 16 22:10:47.054473 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.054139    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="kserve-container"
Jul 16 22:10:47.054473 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.054148    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="419f3623-53f4-46ef-bfea-7f159c51321d" containerName="agent"
Jul 16 22:10:47.057425 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.057398    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.060240 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.060215    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9e75a1-kube-rbac-proxy-sar-config\""
Jul 16 22:10:47.061515 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.061497    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9e75a1-predictor-serving-cert\""
Jul 16 22:10:47.073801 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.073777    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"]
Jul 16 22:10:47.206562 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.206529    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kserve-provision-location\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.206562 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.206575    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec9480e-76e6-4fbd-8e69-a3b620d31168-cabundle-cert\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.206842 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.206655    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.206842 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.206697    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec9480e-76e6-4fbd-8e69-a3b620d31168-proxy-tls\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.206842 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.206737    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdzmm\" (UniqueName: \"kubernetes.io/projected/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kube-api-access-wdzmm\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.307627 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.307527    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdzmm\" (UniqueName: \"kubernetes.io/projected/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kube-api-access-wdzmm\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.307808 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.307651    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kserve-provision-location\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.307808 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.307704    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec9480e-76e6-4fbd-8e69-a3b620d31168-cabundle-cert\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.307808 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.307751    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.307808 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.307770    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec9480e-76e6-4fbd-8e69-a3b620d31168-proxy-tls\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.308145 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.308126    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kserve-provision-location\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.308587 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.308565    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec9480e-76e6-4fbd-8e69-a3b620d31168-cabundle-cert\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.308629 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.308580    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.310317 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.310297    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec9480e-76e6-4fbd-8e69-a3b620d31168-proxy-tls\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.316517 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.316492    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdzmm\" (UniqueName: \"kubernetes.io/projected/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kube-api-access-wdzmm\") pod \"isvc-primary-9e75a1-predictor-7fd479db86-64jt7\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") " pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.368287 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.368258    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:47.522958 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:47.522877    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"]
Jul 16 22:10:47.526711 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:10:47.526680    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec9480e_76e6_4fbd_8e69_a3b620d31168.slice/crio-56dfcd92d527377c7e5768553c77ca889339faa1572d7efef412ad00df2e685e WatchSource:0}: Error finding container 56dfcd92d527377c7e5768553c77ca889339faa1572d7efef412ad00df2e685e: Status 404 returned error can't find the container with id 56dfcd92d527377c7e5768553c77ca889339faa1572d7efef412ad00df2e685e
Jul 16 22:10:48.393933 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:48.393895    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" event={"ID":"8ec9480e-76e6-4fbd-8e69-a3b620d31168","Type":"ContainerStarted","Data":"2acb72bfcd5d506f0e3c3ed57db6752ff0e2b027a29048b2d73b161d1c0b0009"}
Jul 16 22:10:48.394318 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:48.393962    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" event={"ID":"8ec9480e-76e6-4fbd-8e69-a3b620d31168","Type":"ContainerStarted","Data":"56dfcd92d527377c7e5768553c77ca889339faa1572d7efef412ad00df2e685e"}
Jul 16 22:10:49.843573 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:49.843532    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 16 22:10:51.413167 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.413135    2570 generic.go:356] "Generic (PLEG): container finished" podID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerID="0c554e334ecd1763296ec66e63608db491efa196c55bcb4a1bfdcfeaa79c0b41" exitCode=0
Jul 16 22:10:51.413597 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.413212    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" event={"ID":"8f4d1fa8-49c5-454f-9641-c4baaf90fe78","Type":"ContainerDied","Data":"0c554e334ecd1763296ec66e63608db491efa196c55bcb4a1bfdcfeaa79c0b41"}
Jul 16 22:10:51.414740 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.414716    2570 generic.go:356] "Generic (PLEG): container finished" podID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerID="2acb72bfcd5d506f0e3c3ed57db6752ff0e2b027a29048b2d73b161d1c0b0009" exitCode=0
Jul 16 22:10:51.414839 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.414761    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" event={"ID":"8ec9480e-76e6-4fbd-8e69-a3b620d31168","Type":"ContainerDied","Data":"2acb72bfcd5d506f0e3c3ed57db6752ff0e2b027a29048b2d73b161d1c0b0009"}
Jul 16 22:10:51.544681 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.544654    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:10:51.636053 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.635952    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-cabundle-cert\") pod \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") "
Jul 16 22:10:51.636053 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.636002    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wjkz\" (UniqueName: \"kubernetes.io/projected/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kube-api-access-9wjkz\") pod \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") "
Jul 16 22:10:51.636053 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.636051    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kube-rbac-proxy-sar-config\") pod \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") "
Jul 16 22:10:51.636349 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.636107    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kserve-provision-location\") pod \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") "
Jul 16 22:10:51.636349 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.636151    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-proxy-tls\") pod \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\" (UID: \"8f4d1fa8-49c5-454f-9641-c4baaf90fe78\") "
Jul 16 22:10:51.636656 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.636617    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f4d1fa8-49c5-454f-9641-c4baaf90fe78" (UID: "8f4d1fa8-49c5-454f-9641-c4baaf90fe78"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:10:51.636743 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.636679    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f4d1fa8-49c5-454f-9641-c4baaf90fe78" (UID: "8f4d1fa8-49c5-454f-9641-c4baaf90fe78"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:10:51.636743 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.636731    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f4d1fa8-49c5-454f-9641-c4baaf90fe78" (UID: "8f4d1fa8-49c5-454f-9641-c4baaf90fe78"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:10:51.638341 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.638319    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f4d1fa8-49c5-454f-9641-c4baaf90fe78" (UID: "8f4d1fa8-49c5-454f-9641-c4baaf90fe78"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:10:51.638467 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.638447    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kube-api-access-9wjkz" (OuterVolumeSpecName: "kube-api-access-9wjkz") pod "8f4d1fa8-49c5-454f-9641-c4baaf90fe78" (UID: "8f4d1fa8-49c5-454f-9641-c4baaf90fe78"). InnerVolumeSpecName "kube-api-access-9wjkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:10:51.737106 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.737058    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:10:51.737106 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.737109    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kserve-provision-location\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:10:51.737276 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.737119    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:10:51.737276 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.737130    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-cabundle-cert\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:10:51.737276 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:51.737140    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wjkz\" (UniqueName: \"kubernetes.io/projected/8f4d1fa8-49c5-454f-9641-c4baaf90fe78-kube-api-access-9wjkz\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:10:52.421810 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.421721    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" event={"ID":"8ec9480e-76e6-4fbd-8e69-a3b620d31168","Type":"ContainerStarted","Data":"8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb"}
Jul 16 22:10:52.421810 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.421793    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" event={"ID":"8ec9480e-76e6-4fbd-8e69-a3b620d31168","Type":"ContainerStarted","Data":"913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68"}
Jul 16 22:10:52.422277 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.422096    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:52.422277 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.422267    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:52.423765 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.423736    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck" event={"ID":"8f4d1fa8-49c5-454f-9641-c4baaf90fe78","Type":"ContainerDied","Data":"3f3848f7ad5fb3ea3ad6d40d1e1167e130dc8487ff8b15e73c1874a072f12621"}
Jul 16 22:10:52.423889 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.423769    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"
Jul 16 22:10:52.423889 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.423792    2570 scope.go:117] "RemoveContainer" containerID="65714bac0cd1e5ac1897c55e7dc0b7bff425ddc8cf365b43627da78ffaa7ac29"
Jul 16 22:10:52.423995 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.423932    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 16 22:10:52.432020 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.431998    2570 scope.go:117] "RemoveContainer" containerID="0c554e334ecd1763296ec66e63608db491efa196c55bcb4a1bfdcfeaa79c0b41"
Jul 16 22:10:52.438958 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.438937    2570 scope.go:117] "RemoveContainer" containerID="693bc7778d868117174f840161e7a5969f2888281c018b4636088bf966bf7ad7"
Jul 16 22:10:52.443095 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.443041    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podStartSLOduration=1.913322361 podStartE2EDuration="5.4430276s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.529705239s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:10:52.442135158 +0000 UTC m=+1237.039675391" watchObservedRunningTime="2026-07-16 22:10:52.4430276 +0000 UTC m=+1237.040567831"
Jul 16 22:10:52.455290 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.455265    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"]
Jul 16 22:10:52.458929 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:52.458906    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck"]
Jul 16 22:10:53.429049 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:53.429014    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 16 22:10:53.871393 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:53.871355    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" path="/var/lib/kubelet/pods/8f4d1fa8-49c5-454f-9641-c4baaf90fe78/volumes"
Jul 16 22:10:58.433947 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:58.433919    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:10:58.434478 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:10:58.434452    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 16 22:11:08.435248 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:11:08.435167    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 16 22:11:18.435123 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:11:18.435068    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 16 22:11:28.435377 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:11:28.435339    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 16 22:11:38.435013 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:11:38.434975    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 16 22:11:48.434674 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:11:48.434632    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 16 22:11:58.434630 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:11:58.434585    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 16 22:12:08.435187 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:08.435154    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:12:17.321563 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.321528    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"]
Jul 16 22:12:17.321952 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.321913    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="storage-initializer"
Jul 16 22:12:17.321952 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.321928    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="storage-initializer"
Jul 16 22:12:17.321952 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.321947    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container"
Jul 16 22:12:17.322056 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.321955    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container"
Jul 16 22:12:17.322056 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.321969    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kube-rbac-proxy"
Jul 16 22:12:17.322056 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.321978    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kube-rbac-proxy"
Jul 16 22:12:17.322170 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.322072    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kserve-container"
Jul 16 22:12:17.322170 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.322098    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f4d1fa8-49c5-454f-9641-c4baaf90fe78" containerName="kube-rbac-proxy"
Jul 16 22:12:17.325399 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.325372    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.327659 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.327628    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-9e75a1-dockercfg-fcg77\""
Jul 16 22:12:17.327762 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.327740    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9e75a1-predictor-serving-cert\""
Jul 16 22:12:17.328236 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.328208    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-9e75a1\""
Jul 16 22:12:17.328236 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.328209    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9e75a1-kube-rbac-proxy-sar-config\""
Jul 16 22:12:17.337272 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.337251    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"]
Jul 16 22:12:17.446676 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.446654    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a8d798a-ef3b-456a-81a5-a5f4664590a3-cabundle-cert\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.446809 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.446696    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kserve-provision-location\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.446809 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.446721    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4qgl\" (UniqueName: \"kubernetes.io/projected/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kube-api-access-c4qgl\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.446936 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.446815    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a8d798a-ef3b-456a-81a5-a5f4664590a3-proxy-tls\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.446936 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.446859    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.547671 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.547643    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kserve-provision-location\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.547809 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.547678    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4qgl\" (UniqueName: \"kubernetes.io/projected/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kube-api-access-c4qgl\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.547809 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.547696    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a8d798a-ef3b-456a-81a5-a5f4664590a3-proxy-tls\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.547809 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.547723    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.547809 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.547754    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a8d798a-ef3b-456a-81a5-a5f4664590a3-cabundle-cert\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.548139 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.548120    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kserve-provision-location\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.548403 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.548386    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a8d798a-ef3b-456a-81a5-a5f4664590a3-cabundle-cert\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.548456 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.548439    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.550169 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.550151    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a8d798a-ef3b-456a-81a5-a5f4664590a3-proxy-tls\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.555470 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.555450    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4qgl\" (UniqueName: \"kubernetes.io/projected/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kube-api-access-c4qgl\") pod \"isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") " pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.637649 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.637597    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:17.765839 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:17.765814    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"]
Jul 16 22:12:17.767711 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:12:17.767683    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8d798a_ef3b_456a_81a5_a5f4664590a3.slice/crio-9da9039ebbd3e6316f01247564aa19fbb0b0db9ff817c01773fe24028f03a3ee WatchSource:0}: Error finding container 9da9039ebbd3e6316f01247564aa19fbb0b0db9ff817c01773fe24028f03a3ee: Status 404 returned error can't find the container with id 9da9039ebbd3e6316f01247564aa19fbb0b0db9ff817c01773fe24028f03a3ee
Jul 16 22:12:18.729165 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:18.729062    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s" event={"ID":"3a8d798a-ef3b-456a-81a5-a5f4664590a3","Type":"ContainerStarted","Data":"465e5a5e492a9dfc4ce699cb75b1ee6b44e1558e4d0a2c1fe603e65c31023ccf"}
Jul 16 22:12:18.729563 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:18.729174    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s" event={"ID":"3a8d798a-ef3b-456a-81a5-a5f4664590a3","Type":"ContainerStarted","Data":"9da9039ebbd3e6316f01247564aa19fbb0b0db9ff817c01773fe24028f03a3ee"}
Jul 16 22:12:23.747823 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:23.747798    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s_3a8d798a-ef3b-456a-81a5-a5f4664590a3/storage-initializer/0.log"
Jul 16 22:12:23.748236 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:23.747835    2570 generic.go:356] "Generic (PLEG): container finished" podID="3a8d798a-ef3b-456a-81a5-a5f4664590a3" containerID="465e5a5e492a9dfc4ce699cb75b1ee6b44e1558e4d0a2c1fe603e65c31023ccf" exitCode=1
Jul 16 22:12:23.748236 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:23.747870    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s" event={"ID":"3a8d798a-ef3b-456a-81a5-a5f4664590a3","Type":"ContainerDied","Data":"465e5a5e492a9dfc4ce699cb75b1ee6b44e1558e4d0a2c1fe603e65c31023ccf"}
Jul 16 22:12:24.752881 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:24.752854    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s_3a8d798a-ef3b-456a-81a5-a5f4664590a3/storage-initializer/0.log"
Jul 16 22:12:24.753277 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:24.752947    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s" event={"ID":"3a8d798a-ef3b-456a-81a5-a5f4664590a3","Type":"ContainerStarted","Data":"9b5d9124afe1d37dfe2cc6c630aadd184d2ef8f8e549f397965e8aaf5990d16d"}
Jul 16 22:12:28.768918 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:28.768890    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s_3a8d798a-ef3b-456a-81a5-a5f4664590a3/storage-initializer/1.log"
Jul 16 22:12:28.769321 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:28.769306    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s_3a8d798a-ef3b-456a-81a5-a5f4664590a3/storage-initializer/0.log"
Jul 16 22:12:28.769364 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:28.769339    2570 generic.go:356] "Generic (PLEG): container finished" podID="3a8d798a-ef3b-456a-81a5-a5f4664590a3" containerID="9b5d9124afe1d37dfe2cc6c630aadd184d2ef8f8e549f397965e8aaf5990d16d" exitCode=1
Jul 16 22:12:28.769413 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:28.769377    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s" event={"ID":"3a8d798a-ef3b-456a-81a5-a5f4664590a3","Type":"ContainerDied","Data":"9b5d9124afe1d37dfe2cc6c630aadd184d2ef8f8e549f397965e8aaf5990d16d"}
Jul 16 22:12:28.769449 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:28.769439    2570 scope.go:117] "RemoveContainer" containerID="465e5a5e492a9dfc4ce699cb75b1ee6b44e1558e4d0a2c1fe603e65c31023ccf"
Jul 16 22:12:28.769853 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:28.769834    2570 scope.go:117] "RemoveContainer" containerID="465e5a5e492a9dfc4ce699cb75b1ee6b44e1558e4d0a2c1fe603e65c31023ccf"
Jul 16 22:12:28.780054 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:12:28.780033    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s_kserve-ci-e2e-test(3a8d798a-ef3b-456a-81a5-a5f4664590a3)\"" pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s" podUID="3a8d798a-ef3b-456a-81a5-a5f4664590a3"
Jul 16 22:12:28.780224 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:12:28.780200    2570 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s_kserve-ci-e2e-test_3a8d798a-ef3b-456a-81a5-a5f4664590a3_0 in pod sandbox 9da9039ebbd3e6316f01247564aa19fbb0b0db9ff817c01773fe24028f03a3ee from index: no such id: '465e5a5e492a9dfc4ce699cb75b1ee6b44e1558e4d0a2c1fe603e65c31023ccf'" containerID="465e5a5e492a9dfc4ce699cb75b1ee6b44e1558e4d0a2c1fe603e65c31023ccf"
Jul 16 22:12:28.780292 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:28.780233    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465e5a5e492a9dfc4ce699cb75b1ee6b44e1558e4d0a2c1fe603e65c31023ccf"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s_kserve-ci-e2e-test_3a8d798a-ef3b-456a-81a5-a5f4664590a3_0 in pod sandbox 9da9039ebbd3e6316f01247564aa19fbb0b0db9ff817c01773fe24028f03a3ee from index: no such id: '465e5a5e492a9dfc4ce699cb75b1ee6b44e1558e4d0a2c1fe603e65c31023ccf'"
Jul 16 22:12:29.774987 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:29.774956    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s_3a8d798a-ef3b-456a-81a5-a5f4664590a3/storage-initializer/1.log"
Jul 16 22:12:35.330078 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.330045    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"]
Jul 16 22:12:35.390056 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.390030    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"]
Jul 16 22:12:35.390349 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.390301    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container" containerID="cri-o://913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68" gracePeriod=30
Jul 16 22:12:35.390451 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.390380    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kube-rbac-proxy" containerID="cri-o://8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb" gracePeriod=30
Jul 16 22:12:35.508254 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.508234    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s_3a8d798a-ef3b-456a-81a5-a5f4664590a3/storage-initializer/1.log"
Jul 16 22:12:35.510273 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.510258    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:35.674374 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.674297    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4qgl\" (UniqueName: \"kubernetes.io/projected/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kube-api-access-c4qgl\") pod \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") "
Jul 16 22:12:35.674374 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.674359    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a8d798a-ef3b-456a-81a5-a5f4664590a3-proxy-tls\") pod \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") "
Jul 16 22:12:35.674559 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.674410    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a8d798a-ef3b-456a-81a5-a5f4664590a3-cabundle-cert\") pod \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") "
Jul 16 22:12:35.674559 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.674438    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kserve-provision-location\") pod \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") "
Jul 16 22:12:35.674559 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.674475    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kube-rbac-proxy-sar-config\") pod \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\" (UID: \"3a8d798a-ef3b-456a-81a5-a5f4664590a3\") "
Jul 16 22:12:35.674705 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.674677    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8d798a-ef3b-456a-81a5-a5f4664590a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a8d798a-ef3b-456a-81a5-a5f4664590a3" (UID: "3a8d798a-ef3b-456a-81a5-a5f4664590a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:12:35.674770 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.674727    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a8d798a-ef3b-456a-81a5-a5f4664590a3" (UID: "3a8d798a-ef3b-456a-81a5-a5f4664590a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:12:35.674930 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.674905    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a8d798a-ef3b-456a-81a5-a5f4664590a3" (UID: "3a8d798a-ef3b-456a-81a5-a5f4664590a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:12:35.676449 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.676420    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kube-api-access-c4qgl" (OuterVolumeSpecName: "kube-api-access-c4qgl") pod "3a8d798a-ef3b-456a-81a5-a5f4664590a3" (UID: "3a8d798a-ef3b-456a-81a5-a5f4664590a3"). InnerVolumeSpecName "kube-api-access-c4qgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:12:35.676549 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.676459    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a8d798a-ef3b-456a-81a5-a5f4664590a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a8d798a-ef3b-456a-81a5-a5f4664590a3" (UID: "3a8d798a-ef3b-456a-81a5-a5f4664590a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:12:35.775892 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.775870    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:35.775892 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.775891    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4qgl\" (UniqueName: \"kubernetes.io/projected/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kube-api-access-c4qgl\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:35.776017 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.775900    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a8d798a-ef3b-456a-81a5-a5f4664590a3-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:35.776017 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.775911    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a8d798a-ef3b-456a-81a5-a5f4664590a3-cabundle-cert\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:35.776017 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.775920    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a8d798a-ef3b-456a-81a5-a5f4664590a3-kserve-provision-location\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:35.796068 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.796045    2570 generic.go:356] "Generic (PLEG): container finished" podID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerID="8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb" exitCode=2
Jul 16 22:12:35.796189 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.796129    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" event={"ID":"8ec9480e-76e6-4fbd-8e69-a3b620d31168","Type":"ContainerDied","Data":"8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb"}
Jul 16 22:12:35.797351 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.797329    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s_3a8d798a-ef3b-456a-81a5-a5f4664590a3/storage-initializer/1.log"
Jul 16 22:12:35.797449 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.797419    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"
Jul 16 22:12:35.797510 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.797419    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s" event={"ID":"3a8d798a-ef3b-456a-81a5-a5f4664590a3","Type":"ContainerDied","Data":"9da9039ebbd3e6316f01247564aa19fbb0b0db9ff817c01773fe24028f03a3ee"}
Jul 16 22:12:35.797572 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.797558    2570 scope.go:117] "RemoveContainer" containerID="9b5d9124afe1d37dfe2cc6c630aadd184d2ef8f8e549f397965e8aaf5990d16d"
Jul 16 22:12:35.837758 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.837730    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"]
Jul 16 22:12:35.845122 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.845079    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s"]
Jul 16 22:12:35.871347 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:35.871319    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a8d798a-ef3b-456a-81a5-a5f4664590a3" path="/var/lib/kubelet/pods/3a8d798a-ef3b-456a-81a5-a5f4664590a3/volumes"
Jul 16 22:12:38.430136 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:38.430037    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 16 22:12:38.434357 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:38.434332    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 16 22:12:39.326319 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.326296    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:12:39.503545 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.503522    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kserve-provision-location\") pod \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") "
Jul 16 22:12:39.503892 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.503574    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec9480e-76e6-4fbd-8e69-a3b620d31168-cabundle-cert\") pod \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") "
Jul 16 22:12:39.503892 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.503600    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec9480e-76e6-4fbd-8e69-a3b620d31168-proxy-tls\") pod \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") "
Jul 16 22:12:39.503892 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.503628    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kube-rbac-proxy-sar-config\") pod \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") "
Jul 16 22:12:39.503892 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.503648    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdzmm\" (UniqueName: \"kubernetes.io/projected/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kube-api-access-wdzmm\") pod \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\" (UID: \"8ec9480e-76e6-4fbd-8e69-a3b620d31168\") "
Jul 16 22:12:39.503892 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.503807    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ec9480e-76e6-4fbd-8e69-a3b620d31168" (UID: "8ec9480e-76e6-4fbd-8e69-a3b620d31168"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:12:39.504178 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.504053    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec9480e-76e6-4fbd-8e69-a3b620d31168-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ec9480e-76e6-4fbd-8e69-a3b620d31168" (UID: "8ec9480e-76e6-4fbd-8e69-a3b620d31168"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:12:39.504178 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.504122    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ec9480e-76e6-4fbd-8e69-a3b620d31168" (UID: "8ec9480e-76e6-4fbd-8e69-a3b620d31168"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:12:39.505630 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.505606    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec9480e-76e6-4fbd-8e69-a3b620d31168-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ec9480e-76e6-4fbd-8e69-a3b620d31168" (UID: "8ec9480e-76e6-4fbd-8e69-a3b620d31168"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:12:39.505749 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.505731    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kube-api-access-wdzmm" (OuterVolumeSpecName: "kube-api-access-wdzmm") pod "8ec9480e-76e6-4fbd-8e69-a3b620d31168" (UID: "8ec9480e-76e6-4fbd-8e69-a3b620d31168"). InnerVolumeSpecName "kube-api-access-wdzmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:12:39.604323 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.604303    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec9480e-76e6-4fbd-8e69-a3b620d31168-cabundle-cert\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:39.604323 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.604324    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec9480e-76e6-4fbd-8e69-a3b620d31168-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:39.604439 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.604333    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:39.604439 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.604342    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdzmm\" (UniqueName: \"kubernetes.io/projected/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kube-api-access-wdzmm\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:39.604439 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.604350    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec9480e-76e6-4fbd-8e69-a3b620d31168-kserve-provision-location\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:39.815344 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.815286    2570 generic.go:356] "Generic (PLEG): container finished" podID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerID="913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68" exitCode=0
Jul 16 22:12:39.815344 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.815327    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" event={"ID":"8ec9480e-76e6-4fbd-8e69-a3b620d31168","Type":"ContainerDied","Data":"913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68"}
Jul 16 22:12:39.815465 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.815349    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"
Jul 16 22:12:39.815465 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.815383    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7" event={"ID":"8ec9480e-76e6-4fbd-8e69-a3b620d31168","Type":"ContainerDied","Data":"56dfcd92d527377c7e5768553c77ca889339faa1572d7efef412ad00df2e685e"}
Jul 16 22:12:39.815465 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.815414    2570 scope.go:117] "RemoveContainer" containerID="8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb"
Jul 16 22:12:39.823505 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.823482    2570 scope.go:117] "RemoveContainer" containerID="913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68"
Jul 16 22:12:39.830560 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.830544    2570 scope.go:117] "RemoveContainer" containerID="2acb72bfcd5d506f0e3c3ed57db6752ff0e2b027a29048b2d73b161d1c0b0009"
Jul 16 22:12:39.837067 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.837049    2570 scope.go:117] "RemoveContainer" containerID="8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb"
Jul 16 22:12:39.837390 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:12:39.837348    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb\": container with ID starting with 8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb not found: ID does not exist" containerID="8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb"
Jul 16 22:12:39.837443 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.837407    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb"} err="failed to get container status \"8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb\": rpc error: code = NotFound desc = could not find container \"8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb\": container with ID starting with 8ad14b8be835b8f6388893bbee2e44c3741e9a0bf31457d9dfb55f5068da1dfb not found: ID does not exist"
Jul 16 22:12:39.837443 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.837426    2570 scope.go:117] "RemoveContainer" containerID="913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68"
Jul 16 22:12:39.837691 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:12:39.837673    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68\": container with ID starting with 913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68 not found: ID does not exist" containerID="913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68"
Jul 16 22:12:39.837730 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.837697    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68"} err="failed to get container status \"913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68\": rpc error: code = NotFound desc = could not find container \"913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68\": container with ID starting with 913a11934510c5f28d830b06022c3ccee8c9ab8df1ac37c42f2353becc012e68 not found: ID does not exist"
Jul 16 22:12:39.837730 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.837712    2570 scope.go:117] "RemoveContainer" containerID="2acb72bfcd5d506f0e3c3ed57db6752ff0e2b027a29048b2d73b161d1c0b0009"
Jul 16 22:12:39.837937 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:12:39.837920    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2acb72bfcd5d506f0e3c3ed57db6752ff0e2b027a29048b2d73b161d1c0b0009\": container with ID starting with 2acb72bfcd5d506f0e3c3ed57db6752ff0e2b027a29048b2d73b161d1c0b0009 not found: ID does not exist" containerID="2acb72bfcd5d506f0e3c3ed57db6752ff0e2b027a29048b2d73b161d1c0b0009"
Jul 16 22:12:39.837996 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.837946    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acb72bfcd5d506f0e3c3ed57db6752ff0e2b027a29048b2d73b161d1c0b0009"} err="failed to get container status \"2acb72bfcd5d506f0e3c3ed57db6752ff0e2b027a29048b2d73b161d1c0b0009\": rpc error: code = NotFound desc = could not find container \"2acb72bfcd5d506f0e3c3ed57db6752ff0e2b027a29048b2d73b161d1c0b0009\": container with ID starting with 2acb72bfcd5d506f0e3c3ed57db6752ff0e2b027a29048b2d73b161d1c0b0009 not found: ID does not exist"
Jul 16 22:12:39.843111 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.843066    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"]
Jul 16 22:12:39.847306 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.847286    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7"]
Jul 16 22:12:39.871398 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:39.871378    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" path="/var/lib/kubelet/pods/8ec9480e-76e6-4fbd-8e69-a3b620d31168/volumes"
Jul 16 22:12:40.581734 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.581703    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"]
Jul 16 22:12:40.582211 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582072    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a8d798a-ef3b-456a-81a5-a5f4664590a3" containerName="storage-initializer"
Jul 16 22:12:40.582211 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582106    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a8d798a-ef3b-456a-81a5-a5f4664590a3" containerName="storage-initializer"
Jul 16 22:12:40.582211 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582122    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a8d798a-ef3b-456a-81a5-a5f4664590a3" containerName="storage-initializer"
Jul 16 22:12:40.582211 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582129    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a8d798a-ef3b-456a-81a5-a5f4664590a3" containerName="storage-initializer"
Jul 16 22:12:40.582211 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582146    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container"
Jul 16 22:12:40.582211 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582154    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container"
Jul 16 22:12:40.582211 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582168    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kube-rbac-proxy"
Jul 16 22:12:40.582211 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582175    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kube-rbac-proxy"
Jul 16 22:12:40.582211 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582184    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="storage-initializer"
Jul 16 22:12:40.582211 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582192    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="storage-initializer"
Jul 16 22:12:40.582701 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582257    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kserve-container"
Jul 16 22:12:40.582701 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582269    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a8d798a-ef3b-456a-81a5-a5f4664590a3" containerName="storage-initializer"
Jul 16 22:12:40.582701 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582277    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a8d798a-ef3b-456a-81a5-a5f4664590a3" containerName="storage-initializer"
Jul 16 22:12:40.582701 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.582290    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec9480e-76e6-4fbd-8e69-a3b620d31168" containerName="kube-rbac-proxy"
Jul 16 22:12:40.586879 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.586857    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.589188 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.589166    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-279bfc-predictor-serving-cert\""
Jul 16 22:12:40.589300 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.589255    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 22:12:40.589372 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.589325    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-279bfc-dockercfg-tl67q\""
Jul 16 22:12:40.589372 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.589352    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:12:40.589626 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.589611    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-279bfc-kube-rbac-proxy-sar-config\""
Jul 16 22:12:40.589761 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.589746    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-279bfc\""
Jul 16 22:12:40.590124 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.590107    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:12:40.599400 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.599380    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"]
Jul 16 22:12:40.714344 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.714318    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.714472 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.714352    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-proxy-tls\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.714472 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.714414    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-cabundle-cert\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.714472 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.714439    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnsq\" (UniqueName: \"kubernetes.io/projected/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kube-api-access-xgnsq\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.714590 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.714474    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kserve-provision-location\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.815789 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.815754    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnsq\" (UniqueName: \"kubernetes.io/projected/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kube-api-access-xgnsq\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.815875 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.815795    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kserve-provision-location\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.815955 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.815933    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.816016 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.815991    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-proxy-tls\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.816111 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.816066    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-cabundle-cert\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.816249 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.816213    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kserve-provision-location\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.816663 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.816629    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.816737 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.816631    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-cabundle-cert\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.818592 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.818575    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-proxy-tls\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.825844 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.825825    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnsq\" (UniqueName: \"kubernetes.io/projected/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kube-api-access-xgnsq\") pod \"isvc-init-fail-279bfc-predictor-866d647b77-4nw6z\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:40.896714 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:40.896660    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:41.022061 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:41.022034    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"]
Jul 16 22:12:41.024516 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:12:41.024489    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af2e2f7_77a1_4ffd_b104_1904fc64e5f5.slice/crio-2c974946ad54aa5be2026d2ba6cec5505b28d82be2fccb346dea563fb625b907 WatchSource:0}: Error finding container 2c974946ad54aa5be2026d2ba6cec5505b28d82be2fccb346dea563fb625b907: Status 404 returned error can't find the container with id 2c974946ad54aa5be2026d2ba6cec5505b28d82be2fccb346dea563fb625b907
Jul 16 22:12:41.825859 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:41.825827    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z" event={"ID":"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5","Type":"ContainerStarted","Data":"c43af9c39cb64bb59a075bab8114b26e25b7cf60da08d4c3ce810b4cb118f15a"}
Jul 16 22:12:41.826209 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:41.825886    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z" event={"ID":"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5","Type":"ContainerStarted","Data":"2c974946ad54aa5be2026d2ba6cec5505b28d82be2fccb346dea563fb625b907"}
Jul 16 22:12:43.839540 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:43.839510    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-279bfc-predictor-866d647b77-4nw6z_2af2e2f7-77a1-4ffd-b104-1904fc64e5f5/storage-initializer/0.log"
Jul 16 22:12:43.839867 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:43.839547    2570 generic.go:356] "Generic (PLEG): container finished" podID="2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" containerID="c43af9c39cb64bb59a075bab8114b26e25b7cf60da08d4c3ce810b4cb118f15a" exitCode=1
Jul 16 22:12:43.839867 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:43.839583    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z" event={"ID":"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5","Type":"ContainerDied","Data":"c43af9c39cb64bb59a075bab8114b26e25b7cf60da08d4c3ce810b4cb118f15a"}
Jul 16 22:12:44.844745 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:44.844716    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-279bfc-predictor-866d647b77-4nw6z_2af2e2f7-77a1-4ffd-b104-1904fc64e5f5/storage-initializer/0.log"
Jul 16 22:12:44.845197 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:44.844830    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z" event={"ID":"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5","Type":"ContainerStarted","Data":"d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53"}
Jul 16 22:12:45.570739 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:45.570702    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"]
Jul 16 22:12:45.848471 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:45.848382    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z" podUID="2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" containerName="storage-initializer" containerID="cri-o://d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53" gracePeriod=30
Jul 16 22:12:47.780486 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.780463    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-279bfc-predictor-866d647b77-4nw6z_2af2e2f7-77a1-4ffd-b104-1904fc64e5f5/storage-initializer/1.log"
Jul 16 22:12:47.780868 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.780853    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-279bfc-predictor-866d647b77-4nw6z_2af2e2f7-77a1-4ffd-b104-1904fc64e5f5/storage-initializer/0.log"
Jul 16 22:12:47.783196 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.783179    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:47.856156 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.856101    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-279bfc-predictor-866d647b77-4nw6z_2af2e2f7-77a1-4ffd-b104-1904fc64e5f5/storage-initializer/1.log"
Jul 16 22:12:47.856435 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.856415    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-279bfc-predictor-866d647b77-4nw6z_2af2e2f7-77a1-4ffd-b104-1904fc64e5f5/storage-initializer/0.log"
Jul 16 22:12:47.856491 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.856449    2570 generic.go:356] "Generic (PLEG): container finished" podID="2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" containerID="d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53" exitCode=1
Jul 16 22:12:47.856528 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.856503    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"
Jul 16 22:12:47.856565 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.856532    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z" event={"ID":"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5","Type":"ContainerDied","Data":"d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53"}
Jul 16 22:12:47.856600 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.856587    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z" event={"ID":"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5","Type":"ContainerDied","Data":"2c974946ad54aa5be2026d2ba6cec5505b28d82be2fccb346dea563fb625b907"}
Jul 16 22:12:47.856630 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.856616    2570 scope.go:117] "RemoveContainer" containerID="d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53"
Jul 16 22:12:47.863858 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.863843    2570 scope.go:117] "RemoveContainer" containerID="c43af9c39cb64bb59a075bab8114b26e25b7cf60da08d4c3ce810b4cb118f15a"
Jul 16 22:12:47.865854 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.865836    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-cabundle-cert\") pod \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") "
Jul 16 22:12:47.865961 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.865869    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgnsq\" (UniqueName: \"kubernetes.io/projected/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kube-api-access-xgnsq\") pod \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") "
Jul 16 22:12:47.865961 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.865906    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-proxy-tls\") pod \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") "
Jul 16 22:12:47.866068 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.865960    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kserve-provision-location\") pod \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") "
Jul 16 22:12:47.866068 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.865996    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kube-rbac-proxy-sar-config\") pod \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\" (UID: \"2af2e2f7-77a1-4ffd-b104-1904fc64e5f5\") "
Jul 16 22:12:47.866429 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.866245    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" (UID: "2af2e2f7-77a1-4ffd-b104-1904fc64e5f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:12:47.866429 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.866286    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" (UID: "2af2e2f7-77a1-4ffd-b104-1904fc64e5f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:12:47.866593 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.866515    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" (UID: "2af2e2f7-77a1-4ffd-b104-1904fc64e5f5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:12:47.868182 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.868157    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kube-api-access-xgnsq" (OuterVolumeSpecName: "kube-api-access-xgnsq") pod "2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" (UID: "2af2e2f7-77a1-4ffd-b104-1904fc64e5f5"). InnerVolumeSpecName "kube-api-access-xgnsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:12:47.868339 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.868308    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" (UID: "2af2e2f7-77a1-4ffd-b104-1904fc64e5f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:12:47.871685 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.871668    2570 scope.go:117] "RemoveContainer" containerID="d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53"
Jul 16 22:12:47.871960 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:12:47.871935    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53\": container with ID starting with d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53 not found: ID does not exist" containerID="d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53"
Jul 16 22:12:47.872017 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.871973    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53"} err="failed to get container status \"d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53\": rpc error: code = NotFound desc = could not find container \"d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53\": container with ID starting with d208020146f5e8bc96eda7d4427a3aeff7bf88abdc081924c6cd3d838a0a9d53 not found: ID does not exist"
Jul 16 22:12:47.872017 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.871998    2570 scope.go:117] "RemoveContainer" containerID="c43af9c39cb64bb59a075bab8114b26e25b7cf60da08d4c3ce810b4cb118f15a"
Jul 16 22:12:47.872310 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:12:47.872285    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c43af9c39cb64bb59a075bab8114b26e25b7cf60da08d4c3ce810b4cb118f15a\": container with ID starting with c43af9c39cb64bb59a075bab8114b26e25b7cf60da08d4c3ce810b4cb118f15a not found: ID does not exist" containerID="c43af9c39cb64bb59a075bab8114b26e25b7cf60da08d4c3ce810b4cb118f15a"
Jul 16 22:12:47.872353 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.872318    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c43af9c39cb64bb59a075bab8114b26e25b7cf60da08d4c3ce810b4cb118f15a"} err="failed to get container status \"c43af9c39cb64bb59a075bab8114b26e25b7cf60da08d4c3ce810b4cb118f15a\": rpc error: code = NotFound desc = could not find container \"c43af9c39cb64bb59a075bab8114b26e25b7cf60da08d4c3ce810b4cb118f15a\": container with ID starting with c43af9c39cb64bb59a075bab8114b26e25b7cf60da08d4c3ce810b4cb118f15a not found: ID does not exist"
Jul 16 22:12:47.967193 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.967166    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kserve-provision-location\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:47.967193 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.967190    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:47.967332 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.967199    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-cabundle-cert\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:47.967332 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.967209    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgnsq\" (UniqueName: \"kubernetes.io/projected/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-kube-api-access-xgnsq\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:47.967332 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:47.967217    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:12:48.189460 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:48.189388    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"]
Jul 16 22:12:48.194064 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:48.194036    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z"]
Jul 16 22:12:49.871461 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:49.871419    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" path="/var/lib/kubelet/pods/2af2e2f7-77a1-4ffd-b104-1904fc64e5f5/volumes"
Jul 16 22:12:50.819535 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.819496    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"]
Jul 16 22:12:50.819851 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.819837    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" containerName="storage-initializer"
Jul 16 22:12:50.819905 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.819854    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" containerName="storage-initializer"
Jul 16 22:12:50.819905 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.819873    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" containerName="storage-initializer"
Jul 16 22:12:50.819905 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.819881    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" containerName="storage-initializer"
Jul 16 22:12:50.819996 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.819955    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" containerName="storage-initializer"
Jul 16 22:12:50.819996 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.819964    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af2e2f7-77a1-4ffd-b104-1904fc64e5f5" containerName="storage-initializer"
Jul 16 22:12:50.824740 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.824721    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.827971 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.827948    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:12:50.827971 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.827962    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 22:12:50.828200 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.827983    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-3f2fd-predictor-serving-cert\""
Jul 16 22:12:50.828200 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.828041    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xdcsv\""
Jul 16 22:12:50.828786 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.828768    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:12:50.828884 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.828793    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-3f2fd-kube-rbac-proxy-sar-config\""
Jul 16 22:12:50.828884 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.828839    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:12:50.840254 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.840231    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"]
Jul 16 22:12:50.887390 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.887365    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.887735 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.887398    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kserve-provision-location\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.887735 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.887471    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmvnh\" (UniqueName: \"kubernetes.io/projected/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kube-api-access-xmvnh\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.887735 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.887520    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-cabundle-cert\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.887735 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.887551    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-proxy-tls\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.988105 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.988063    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-cabundle-cert\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.988221 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.988118    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-proxy-tls\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.988221 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.988146    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.988221 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.988164    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kserve-provision-location\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.988221 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.988202    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmvnh\" (UniqueName: \"kubernetes.io/projected/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kube-api-access-xmvnh\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.988606 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.988580    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kserve-provision-location\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.988769 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.988750    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.988827 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.988752    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-cabundle-cert\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.990603 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.990585    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-proxy-tls\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:50.997363 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:50.997340    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmvnh\" (UniqueName: \"kubernetes.io/projected/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kube-api-access-xmvnh\") pod \"raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") " pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:51.135138 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:51.135073    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:51.259455 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:51.259426    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"]
Jul 16 22:12:51.262396 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:12:51.262363    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6176d773_9cc1_4c8f_acdb_73ff343a2b2c.slice/crio-79d1aead534c978f8032321f683f1abaef99413e2c736da93f5a68117b64187b WatchSource:0}: Error finding container 79d1aead534c978f8032321f683f1abaef99413e2c736da93f5a68117b64187b: Status 404 returned error can't find the container with id 79d1aead534c978f8032321f683f1abaef99413e2c736da93f5a68117b64187b
Jul 16 22:12:51.872628 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:51.872593    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" event={"ID":"6176d773-9cc1-4c8f-acdb-73ff343a2b2c","Type":"ContainerStarted","Data":"1fa5258495edf281469533ac098badaca985665efac5043bf503b9f5dba5fe91"}
Jul 16 22:12:51.872793 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:51.872660    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" event={"ID":"6176d773-9cc1-4c8f-acdb-73ff343a2b2c","Type":"ContainerStarted","Data":"79d1aead534c978f8032321f683f1abaef99413e2c736da93f5a68117b64187b"}
Jul 16 22:12:55.891219 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:55.891184    2570 generic.go:356] "Generic (PLEG): container finished" podID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerID="1fa5258495edf281469533ac098badaca985665efac5043bf503b9f5dba5fe91" exitCode=0
Jul 16 22:12:55.891616 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:55.891236    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" event={"ID":"6176d773-9cc1-4c8f-acdb-73ff343a2b2c","Type":"ContainerDied","Data":"1fa5258495edf281469533ac098badaca985665efac5043bf503b9f5dba5fe91"}
Jul 16 22:12:56.897270 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:56.897232    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" event={"ID":"6176d773-9cc1-4c8f-acdb-73ff343a2b2c","Type":"ContainerStarted","Data":"7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810"}
Jul 16 22:12:56.897667 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:56.897300    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" event={"ID":"6176d773-9cc1-4c8f-acdb-73ff343a2b2c","Type":"ContainerStarted","Data":"8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986"}
Jul 16 22:12:56.897667 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:56.897505    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:56.919925 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:56.919857    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podStartSLOduration=3.239761653 podStartE2EDuration="6.91984397s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.680082317s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:12:56.918871352 +0000 UTC m=+1361.516411584" watchObservedRunningTime="2026-07-16 22:12:56.91984397 +0000 UTC m=+1361.517384203"
Jul 16 22:12:57.900806 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:57.900772    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:12:57.901802 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:57.901778    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 22:12:58.903892 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:12:58.903851    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 22:13:03.909308 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:13:03.909278    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:13:03.909795 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:13:03.909766    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 22:13:13.909886 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:13:13.909846    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 22:13:23.910471 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:13:23.910432    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 22:13:33.910479 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:13:33.910438    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 22:13:43.909894 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:13:43.909860    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 22:13:53.910553 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:13:53.910516    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 22:14:03.910541 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:03.910503    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 22:14:13.911134 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:13.911061    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:14:20.946558 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:20.946520    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"]
Jul 16 22:14:20.946957 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:20.946853    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container" containerID="cri-o://8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986" gracePeriod=30
Jul 16 22:14:20.946957 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:20.946899    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kube-rbac-proxy" containerID="cri-o://7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810" gracePeriod=30
Jul 16 22:14:21.199735 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:21.199673    2570 generic.go:356] "Generic (PLEG): container finished" podID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerID="7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810" exitCode=2
Jul 16 22:14:21.199833 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:21.199741    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" event={"ID":"6176d773-9cc1-4c8f-acdb-73ff343a2b2c","Type":"ContainerDied","Data":"7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810"}
Jul 16 22:14:23.904747 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:23.904708    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 16 22:14:23.910194 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:23.910164    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 22:14:24.780162 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:24.780134    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:14:24.926738 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:24.926662    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmvnh\" (UniqueName: \"kubernetes.io/projected/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kube-api-access-xmvnh\") pod \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") "
Jul 16 22:14:24.926738 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:24.926714    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kserve-provision-location\") pod \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") "
Jul 16 22:14:24.926738 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:24.926736    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-proxy-tls\") pod \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") "
Jul 16 22:14:24.927205 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:24.926755    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-cabundle-cert\") pod \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") "
Jul 16 22:14:24.927205 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:24.926798    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kube-rbac-proxy-sar-config\") pod \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\" (UID: \"6176d773-9cc1-4c8f-acdb-73ff343a2b2c\") "
Jul 16 22:14:24.927205 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:24.927145    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6176d773-9cc1-4c8f-acdb-73ff343a2b2c" (UID: "6176d773-9cc1-4c8f-acdb-73ff343a2b2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:24.927205 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:24.927156    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6176d773-9cc1-4c8f-acdb-73ff343a2b2c" (UID: "6176d773-9cc1-4c8f-acdb-73ff343a2b2c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:14:24.927205 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:24.927164    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6176d773-9cc1-4c8f-acdb-73ff343a2b2c" (UID: "6176d773-9cc1-4c8f-acdb-73ff343a2b2c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:14:24.928806 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:24.928777    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6176d773-9cc1-4c8f-acdb-73ff343a2b2c" (UID: "6176d773-9cc1-4c8f-acdb-73ff343a2b2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:14:24.928905 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:24.928880    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kube-api-access-xmvnh" (OuterVolumeSpecName: "kube-api-access-xmvnh") pod "6176d773-9cc1-4c8f-acdb-73ff343a2b2c" (UID: "6176d773-9cc1-4c8f-acdb-73ff343a2b2c"). InnerVolumeSpecName "kube-api-access-xmvnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:14:25.030641 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.030608    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmvnh\" (UniqueName: \"kubernetes.io/projected/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kube-api-access-xmvnh\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:14:25.030755 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.030644    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kserve-provision-location\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:14:25.030755 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.030670    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:14:25.030855 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.030838    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-cabundle-cert\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:14:25.030900 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.030865    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6176d773-9cc1-4c8f-acdb-73ff343a2b2c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:14:25.215435 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.215367    2570 generic.go:356] "Generic (PLEG): container finished" podID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerID="8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986" exitCode=0
Jul 16 22:14:25.215435 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.215413    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" event={"ID":"6176d773-9cc1-4c8f-acdb-73ff343a2b2c","Type":"ContainerDied","Data":"8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986"}
Jul 16 22:14:25.215571 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.215440    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"
Jul 16 22:14:25.215571 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.215501    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk" event={"ID":"6176d773-9cc1-4c8f-acdb-73ff343a2b2c","Type":"ContainerDied","Data":"79d1aead534c978f8032321f683f1abaef99413e2c736da93f5a68117b64187b"}
Jul 16 22:14:25.215571 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.215532    2570 scope.go:117] "RemoveContainer" containerID="7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810"
Jul 16 22:14:25.224052 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.224029    2570 scope.go:117] "RemoveContainer" containerID="8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986"
Jul 16 22:14:25.233276 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.233259    2570 scope.go:117] "RemoveContainer" containerID="1fa5258495edf281469533ac098badaca985665efac5043bf503b9f5dba5fe91"
Jul 16 22:14:25.236009 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.235986    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"]
Jul 16 22:14:25.238592 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.238572    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk"]
Jul 16 22:14:25.240919 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.240885    2570 scope.go:117] "RemoveContainer" containerID="7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810"
Jul 16 22:14:25.241251 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:14:25.241230    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810\": container with ID starting with 7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810 not found: ID does not exist" containerID="7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810"
Jul 16 22:14:25.241314 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.241258    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810"} err="failed to get container status \"7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810\": rpc error: code = NotFound desc = could not find container \"7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810\": container with ID starting with 7c8b71746063bc049f1bacdd57f3c3dd2a7e009446e0d11de23b89bf2fd66810 not found: ID does not exist"
Jul 16 22:14:25.241314 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.241276    2570 scope.go:117] "RemoveContainer" containerID="8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986"
Jul 16 22:14:25.241550 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:14:25.241514    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986\": container with ID starting with 8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986 not found: ID does not exist" containerID="8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986"
Jul 16 22:14:25.241621 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.241561    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986"} err="failed to get container status \"8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986\": rpc error: code = NotFound desc = could not find container \"8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986\": container with ID starting with 8a29b69ba6e3ef95a2927290bd3ef6dbb8b7dc42f536834f45611d80c3676986 not found: ID does not exist"
Jul 16 22:14:25.241621 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.241584    2570 scope.go:117] "RemoveContainer" containerID="1fa5258495edf281469533ac098badaca985665efac5043bf503b9f5dba5fe91"
Jul 16 22:14:25.241855 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:14:25.241838    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa5258495edf281469533ac098badaca985665efac5043bf503b9f5dba5fe91\": container with ID starting with 1fa5258495edf281469533ac098badaca985665efac5043bf503b9f5dba5fe91 not found: ID does not exist" containerID="1fa5258495edf281469533ac098badaca985665efac5043bf503b9f5dba5fe91"
Jul 16 22:14:25.241895 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.241861    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa5258495edf281469533ac098badaca985665efac5043bf503b9f5dba5fe91"} err="failed to get container status \"1fa5258495edf281469533ac098badaca985665efac5043bf503b9f5dba5fe91\": rpc error: code = NotFound desc = could not find container \"1fa5258495edf281469533ac098badaca985665efac5043bf503b9f5dba5fe91\": container with ID starting with 1fa5258495edf281469533ac098badaca985665efac5043bf503b9f5dba5fe91 not found: ID does not exist"
Jul 16 22:14:25.870997 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:25.870961    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" path="/var/lib/kubelet/pods/6176d773-9cc1-4c8f-acdb-73ff343a2b2c/volumes"
Jul 16 22:14:26.159029 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.158897    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"]
Jul 16 22:14:26.159458 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.159152    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container"
Jul 16 22:14:26.159458 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.159163    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container"
Jul 16 22:14:26.159458 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.159174    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="storage-initializer"
Jul 16 22:14:26.159458 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.159180    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="storage-initializer"
Jul 16 22:14:26.159458 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.159196    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kube-rbac-proxy"
Jul 16 22:14:26.159458 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.159201    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kube-rbac-proxy"
Jul 16 22:14:26.159458 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.159241    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kserve-container"
Jul 16 22:14:26.159458 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.159249    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6176d773-9cc1-4c8f-acdb-73ff343a2b2c" containerName="kube-rbac-proxy"
Jul 16 22:14:26.163576 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.163560    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.165890 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.165865    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:14:26.166019 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.165912    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-24151-kube-rbac-proxy-sar-config\""
Jul 16 22:14:26.166019 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.165924    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:14:26.166019 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.165974    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-24151-predictor-serving-cert\""
Jul 16 22:14:26.166019 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.165989    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:14:26.166298 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.166284    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 22:14:26.166844 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.166829    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xdcsv\""
Jul 16 22:14:26.171778 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.171758    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"]
Jul 16 22:14:26.239119 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.239095    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97c970ba-3dcc-4735-84c3-401acc97f70a-kserve-provision-location\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.239232 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.239141    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97c970ba-3dcc-4735-84c3-401acc97f70a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.239232 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.239169    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97c970ba-3dcc-4735-84c3-401acc97f70a-proxy-tls\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.239232 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.239185    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97c970ba-3dcc-4735-84c3-401acc97f70a-cabundle-cert\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.239232 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.239214    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fg8r\" (UniqueName: \"kubernetes.io/projected/97c970ba-3dcc-4735-84c3-401acc97f70a-kube-api-access-7fg8r\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.340406 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.340381    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97c970ba-3dcc-4735-84c3-401acc97f70a-kserve-provision-location\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.340509 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.340419    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97c970ba-3dcc-4735-84c3-401acc97f70a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.340509 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.340448    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97c970ba-3dcc-4735-84c3-401acc97f70a-proxy-tls\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.340633 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.340618    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97c970ba-3dcc-4735-84c3-401acc97f70a-cabundle-cert\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.340682 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.340672    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fg8r\" (UniqueName: \"kubernetes.io/projected/97c970ba-3dcc-4735-84c3-401acc97f70a-kube-api-access-7fg8r\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.340792 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.340773    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97c970ba-3dcc-4735-84c3-401acc97f70a-kserve-provision-location\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.341173 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.341148    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97c970ba-3dcc-4735-84c3-401acc97f70a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.341173 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.341162    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97c970ba-3dcc-4735-84c3-401acc97f70a-cabundle-cert\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.342947 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.342927    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97c970ba-3dcc-4735-84c3-401acc97f70a-proxy-tls\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.349534 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.349508    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fg8r\" (UniqueName: \"kubernetes.io/projected/97c970ba-3dcc-4735-84c3-401acc97f70a-kube-api-access-7fg8r\") pod \"raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.474812 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.474786    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:26.612152 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:26.612123    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"]
Jul 16 22:14:26.615533 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:14:26.615502    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c970ba_3dcc_4735_84c3_401acc97f70a.slice/crio-f99ac23dbb0c20204ad50c42072c290d38ce8e2c2c36bce62b97b6e173c1358f WatchSource:0}: Error finding container f99ac23dbb0c20204ad50c42072c290d38ce8e2c2c36bce62b97b6e173c1358f: Status 404 returned error can't find the container with id f99ac23dbb0c20204ad50c42072c290d38ce8e2c2c36bce62b97b6e173c1358f
Jul 16 22:14:27.224480 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:27.224447    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" event={"ID":"97c970ba-3dcc-4735-84c3-401acc97f70a","Type":"ContainerStarted","Data":"3d4d846ac2c9417ff5312f1944d2ab078c943e63e5f2506b12c542438ba3cef3"}
Jul 16 22:14:27.224831 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:27.224508    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" event={"ID":"97c970ba-3dcc-4735-84c3-401acc97f70a","Type":"ContainerStarted","Data":"f99ac23dbb0c20204ad50c42072c290d38ce8e2c2c36bce62b97b6e173c1358f"}
Jul 16 22:14:31.241002 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:31.240958    2570 generic.go:356] "Generic (PLEG): container finished" podID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerID="3d4d846ac2c9417ff5312f1944d2ab078c943e63e5f2506b12c542438ba3cef3" exitCode=0
Jul 16 22:14:31.241464 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:31.241034    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" event={"ID":"97c970ba-3dcc-4735-84c3-401acc97f70a","Type":"ContainerDied","Data":"3d4d846ac2c9417ff5312f1944d2ab078c943e63e5f2506b12c542438ba3cef3"}
Jul 16 22:14:32.246829 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:32.246794    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" event={"ID":"97c970ba-3dcc-4735-84c3-401acc97f70a","Type":"ContainerStarted","Data":"018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938"}
Jul 16 22:14:32.247211 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:32.246855    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" event={"ID":"97c970ba-3dcc-4735-84c3-401acc97f70a","Type":"ContainerStarted","Data":"61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11"}
Jul 16 22:14:32.247211 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:32.247051    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:32.270191 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:32.270145    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podStartSLOduration=2.558544802 podStartE2EDuration="6.270131696s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.711586894s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:14:32.268854515 +0000 UTC m=+1456.866394746" watchObservedRunningTime="2026-07-16 22:14:32.270131696 +0000 UTC m=+1456.867671929"
Jul 16 22:14:33.250892 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:33.250860    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:33.252218 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:33.252187    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 16 22:14:34.253866 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:34.253825    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 16 22:14:39.258839 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:39.258807    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:14:39.259279 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:39.259254    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 16 22:14:49.259879 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:49.259834    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 16 22:14:59.259483 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:14:59.259438    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 16 22:15:09.259727 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:09.259681    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 16 22:15:15.853385 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:15.853356    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 22:15:15.856585 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:15.856563    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 22:15:19.259391 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:19.259348    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 16 22:15:29.259474 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:29.259433    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 16 22:15:39.259728 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:39.259642    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 16 22:15:49.260749 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:49.260720    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:15:56.284644 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:56.284615    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"]
Jul 16 22:15:56.285159 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:56.284879    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container" containerID="cri-o://61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11" gracePeriod=30
Jul 16 22:15:56.285159 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:56.284900    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kube-rbac-proxy" containerID="cri-o://018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938" gracePeriod=30
Jul 16 22:15:56.554491 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:56.554430    2570 generic.go:356] "Generic (PLEG): container finished" podID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerID="018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938" exitCode=2
Jul 16 22:15:56.554491 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:56.554471    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" event={"ID":"97c970ba-3dcc-4735-84c3-401acc97f70a","Type":"ContainerDied","Data":"018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938"}
Jul 16 22:15:59.254035 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:59.253997    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 16 22:15:59.259202 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:15:59.259175    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 16 22:16:00.225305 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.225283    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:16:00.307186 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.307130    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97c970ba-3dcc-4735-84c3-401acc97f70a-cabundle-cert\") pod \"97c970ba-3dcc-4735-84c3-401acc97f70a\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") "
Jul 16 22:16:00.307186 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.307166    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97c970ba-3dcc-4735-84c3-401acc97f70a-proxy-tls\") pod \"97c970ba-3dcc-4735-84c3-401acc97f70a\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") "
Jul 16 22:16:00.307581 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.307223    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97c970ba-3dcc-4735-84c3-401acc97f70a-kserve-provision-location\") pod \"97c970ba-3dcc-4735-84c3-401acc97f70a\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") "
Jul 16 22:16:00.307581 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.307277    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97c970ba-3dcc-4735-84c3-401acc97f70a-kube-rbac-proxy-sar-config\") pod \"97c970ba-3dcc-4735-84c3-401acc97f70a\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") "
Jul 16 22:16:00.307581 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.307314    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fg8r\" (UniqueName: \"kubernetes.io/projected/97c970ba-3dcc-4735-84c3-401acc97f70a-kube-api-access-7fg8r\") pod \"97c970ba-3dcc-4735-84c3-401acc97f70a\" (UID: \"97c970ba-3dcc-4735-84c3-401acc97f70a\") "
Jul 16 22:16:00.307581 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.307502    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c970ba-3dcc-4735-84c3-401acc97f70a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "97c970ba-3dcc-4735-84c3-401acc97f70a" (UID: "97c970ba-3dcc-4735-84c3-401acc97f70a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:16:00.307766 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.307525    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c970ba-3dcc-4735-84c3-401acc97f70a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "97c970ba-3dcc-4735-84c3-401acc97f70a" (UID: "97c970ba-3dcc-4735-84c3-401acc97f70a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:16:00.307766 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.307679    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c970ba-3dcc-4735-84c3-401acc97f70a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "97c970ba-3dcc-4735-84c3-401acc97f70a" (UID: "97c970ba-3dcc-4735-84c3-401acc97f70a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:16:00.309265 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.309245    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c970ba-3dcc-4735-84c3-401acc97f70a-kube-api-access-7fg8r" (OuterVolumeSpecName: "kube-api-access-7fg8r") pod "97c970ba-3dcc-4735-84c3-401acc97f70a" (UID: "97c970ba-3dcc-4735-84c3-401acc97f70a"). InnerVolumeSpecName "kube-api-access-7fg8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:16:00.309370 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.309355    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c970ba-3dcc-4735-84c3-401acc97f70a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97c970ba-3dcc-4735-84c3-401acc97f70a" (UID: "97c970ba-3dcc-4735-84c3-401acc97f70a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:16:00.408443 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.408422    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fg8r\" (UniqueName: \"kubernetes.io/projected/97c970ba-3dcc-4735-84c3-401acc97f70a-kube-api-access-7fg8r\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:16:00.408443 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.408441    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97c970ba-3dcc-4735-84c3-401acc97f70a-cabundle-cert\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:16:00.408563 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.408451    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97c970ba-3dcc-4735-84c3-401acc97f70a-proxy-tls\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:16:00.408563 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.408459    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97c970ba-3dcc-4735-84c3-401acc97f70a-kserve-provision-location\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:16:00.408563 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.408467    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97c970ba-3dcc-4735-84c3-401acc97f70a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-85.ec2.internal\" DevicePath \"\""
Jul 16 22:16:00.569717 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.569655    2570 generic.go:356] "Generic (PLEG): container finished" podID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerID="61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11" exitCode=0
Jul 16 22:16:00.569963 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.569715    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"
Jul 16 22:16:00.569963 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.569751    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" event={"ID":"97c970ba-3dcc-4735-84c3-401acc97f70a","Type":"ContainerDied","Data":"61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11"}
Jul 16 22:16:00.569963 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.569807    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp" event={"ID":"97c970ba-3dcc-4735-84c3-401acc97f70a","Type":"ContainerDied","Data":"f99ac23dbb0c20204ad50c42072c290d38ce8e2c2c36bce62b97b6e173c1358f"}
Jul 16 22:16:00.569963 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.569832    2570 scope.go:117] "RemoveContainer" containerID="018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938"
Jul 16 22:16:00.577657 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.577642    2570 scope.go:117] "RemoveContainer" containerID="61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11"
Jul 16 22:16:00.584627 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.584610    2570 scope.go:117] "RemoveContainer" containerID="3d4d846ac2c9417ff5312f1944d2ab078c943e63e5f2506b12c542438ba3cef3"
Jul 16 22:16:00.589647 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.589623    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"]
Jul 16 22:16:00.591721 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.591699    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp"]
Jul 16 22:16:00.592233 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.592217    2570 scope.go:117] "RemoveContainer" containerID="018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938"
Jul 16 22:16:00.592491 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:16:00.592476    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938\": container with ID starting with 018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938 not found: ID does not exist" containerID="018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938"
Jul 16 22:16:00.592529 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.592499    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938"} err="failed to get container status \"018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938\": rpc error: code = NotFound desc = could not find container \"018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938\": container with ID starting with 018f0d5b18d5ea25b32ea619dd42e03d8bfac8285707786039106902bc0b6938 not found: ID does not exist"
Jul 16 22:16:00.592529 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.592515    2570 scope.go:117] "RemoveContainer" containerID="61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11"
Jul 16 22:16:00.592747 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:16:00.592730    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11\": container with ID starting with 61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11 not found: ID does not exist" containerID="61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11"
Jul 16 22:16:00.592784 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.592754    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11"} err="failed to get container status \"61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11\": rpc error: code = NotFound desc = could not find container \"61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11\": container with ID starting with 61814dd945afa38619ffb216565336f814911d551dfab7b65201fbac97bcee11 not found: ID does not exist"
Jul 16 22:16:00.592784 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.592771    2570 scope.go:117] "RemoveContainer" containerID="3d4d846ac2c9417ff5312f1944d2ab078c943e63e5f2506b12c542438ba3cef3"
Jul 16 22:16:00.592972 ip-10-0-137-85 kubenswrapper[2570]: E0716 22:16:00.592957    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4d846ac2c9417ff5312f1944d2ab078c943e63e5f2506b12c542438ba3cef3\": container with ID starting with 3d4d846ac2c9417ff5312f1944d2ab078c943e63e5f2506b12c542438ba3cef3 not found: ID does not exist" containerID="3d4d846ac2c9417ff5312f1944d2ab078c943e63e5f2506b12c542438ba3cef3"
Jul 16 22:16:00.593008 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:00.592977    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4d846ac2c9417ff5312f1944d2ab078c943e63e5f2506b12c542438ba3cef3"} err="failed to get container status \"3d4d846ac2c9417ff5312f1944d2ab078c943e63e5f2506b12c542438ba3cef3\": rpc error: code = NotFound desc = could not find container \"3d4d846ac2c9417ff5312f1944d2ab078c943e63e5f2506b12c542438ba3cef3\": container with ID starting with 3d4d846ac2c9417ff5312f1944d2ab078c943e63e5f2506b12c542438ba3cef3 not found: ID does not exist"
Jul 16 22:16:01.871202 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:01.871170    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" path="/var/lib/kubelet/pods/97c970ba-3dcc-4735-84c3-401acc97f70a/volumes"
Jul 16 22:16:28.994037 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:28.993999    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k8wws/must-gather-hp9q7"]
Jul 16 22:16:28.994520 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:28.994270    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container"
Jul 16 22:16:28.994520 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:28.994282    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container"
Jul 16 22:16:28.994520 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:28.994298    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="storage-initializer"
Jul 16 22:16:28.994520 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:28.994303    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="storage-initializer"
Jul 16 22:16:28.994520 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:28.994319    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kube-rbac-proxy"
Jul 16 22:16:28.994520 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:28.994324    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kube-rbac-proxy"
Jul 16 22:16:28.994520 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:28.994366    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kube-rbac-proxy"
Jul 16 22:16:28.994520 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:28.994374    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c970ba-3dcc-4735-84c3-401acc97f70a" containerName="kserve-container"
Jul 16 22:16:28.997198 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:28.997181    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8wws/must-gather-hp9q7"
Jul 16 22:16:28.999879 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:28.999852    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k8wws\"/\"kube-root-ca.crt\""
Jul 16 22:16:29.000021 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:28.999893    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k8wws\"/\"openshift-service-ca.crt\""
Jul 16 22:16:29.000762 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:29.000723    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k8wws\"/\"default-dockercfg-nqlp4\""
Jul 16 22:16:29.005391 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:29.005364    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8wws/must-gather-hp9q7"]
Jul 16 22:16:29.105071 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:29.105044    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e49015fd-dde5-4626-a42d-8a2939001d70-must-gather-output\") pod \"must-gather-hp9q7\" (UID: \"e49015fd-dde5-4626-a42d-8a2939001d70\") " pod="openshift-must-gather-k8wws/must-gather-hp9q7"
Jul 16 22:16:29.105191 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:29.105103    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l4ms\" (UniqueName: \"kubernetes.io/projected/e49015fd-dde5-4626-a42d-8a2939001d70-kube-api-access-6l4ms\") pod \"must-gather-hp9q7\" (UID: \"e49015fd-dde5-4626-a42d-8a2939001d70\") " pod="openshift-must-gather-k8wws/must-gather-hp9q7"
Jul 16 22:16:29.205718 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:29.205688    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e49015fd-dde5-4626-a42d-8a2939001d70-must-gather-output\") pod \"must-gather-hp9q7\" (UID: \"e49015fd-dde5-4626-a42d-8a2939001d70\") " pod="openshift-must-gather-k8wws/must-gather-hp9q7"
Jul 16 22:16:29.205834 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:29.205728    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l4ms\" (UniqueName: \"kubernetes.io/projected/e49015fd-dde5-4626-a42d-8a2939001d70-kube-api-access-6l4ms\") pod \"must-gather-hp9q7\" (UID: \"e49015fd-dde5-4626-a42d-8a2939001d70\") " pod="openshift-must-gather-k8wws/must-gather-hp9q7"
Jul 16 22:16:29.206046 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:29.206024    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e49015fd-dde5-4626-a42d-8a2939001d70-must-gather-output\") pod \"must-gather-hp9q7\" (UID: \"e49015fd-dde5-4626-a42d-8a2939001d70\") " pod="openshift-must-gather-k8wws/must-gather-hp9q7"
Jul 16 22:16:29.214197 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:29.214165    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l4ms\" (UniqueName: \"kubernetes.io/projected/e49015fd-dde5-4626-a42d-8a2939001d70-kube-api-access-6l4ms\") pod \"must-gather-hp9q7\" (UID: \"e49015fd-dde5-4626-a42d-8a2939001d70\") " pod="openshift-must-gather-k8wws/must-gather-hp9q7"
Jul 16 22:16:29.307462 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:29.307409    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8wws/must-gather-hp9q7"
Jul 16 22:16:29.429994 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:29.429959    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8wws/must-gather-hp9q7"]
Jul 16 22:16:29.432928 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:16:29.432897    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode49015fd_dde5_4626_a42d_8a2939001d70.slice/crio-1a3e4196b4498d587ac10ca7a729cfd90b4d3e763ff48d2036043df7dda15b87 WatchSource:0}: Error finding container 1a3e4196b4498d587ac10ca7a729cfd90b4d3e763ff48d2036043df7dda15b87: Status 404 returned error can't find the container with id 1a3e4196b4498d587ac10ca7a729cfd90b4d3e763ff48d2036043df7dda15b87
Jul 16 22:16:29.434790 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:29.434771    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:16:29.673166 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:29.673072    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8wws/must-gather-hp9q7" event={"ID":"e49015fd-dde5-4626-a42d-8a2939001d70","Type":"ContainerStarted","Data":"1a3e4196b4498d587ac10ca7a729cfd90b4d3e763ff48d2036043df7dda15b87"}
Jul 16 22:16:30.680408 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:30.679610    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8wws/must-gather-hp9q7" event={"ID":"e49015fd-dde5-4626-a42d-8a2939001d70","Type":"ContainerStarted","Data":"0075885e0112be50d9ec350d1ef631f38e0c7441e6043f14513fa76e97258ee8"}
Jul 16 22:16:30.680408 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:30.679664    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8wws/must-gather-hp9q7" event={"ID":"e49015fd-dde5-4626-a42d-8a2939001d70","Type":"ContainerStarted","Data":"7d3a1255c443b732ea09bb32d2a4056eeddcc194e5f6b1c977ff7da830f748e4"}
Jul 16 22:16:30.696385 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:30.696323    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k8wws/must-gather-hp9q7" podStartSLOduration=1.798996939 podStartE2EDuration="2.696301297s" totalImagesPullingTime="897.304358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:16:30.695873718 +0000 UTC m=+1575.293413950" watchObservedRunningTime="2026-07-16 22:16:30.696301297 +0000 UTC m=+1575.293841531"
Jul 16 22:16:32.004384 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:32.004355    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m7rl6_cbbaa953-ab0b-4594-a449-06fab72528f5/global-pull-secret-syncer/0.log"
Jul 16 22:16:32.170240 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:32.170207    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xn4rj_4ac12a88-9208-4759-9c96-3e5de06602a3/konnectivity-agent/0.log"
Jul 16 22:16:32.215934 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:32.215896    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-85.ec2.internal_186a905c8a5ab6ff537de73d191d65e4/haproxy/0.log"
Jul 16 22:16:36.803944 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:36.803914    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-phmhr_29cee3ea-c23f-454a-9ea1-168e571b88bf/kube-state-metrics/0.log"
Jul 16 22:16:36.846041 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:36.845991    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-phmhr_29cee3ea-c23f-454a-9ea1-168e571b88bf/kube-rbac-proxy-main/0.log"
Jul 16 22:16:36.887046 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:36.886966    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-phmhr_29cee3ea-c23f-454a-9ea1-168e571b88bf/kube-rbac-proxy-self/0.log"
Jul 16 22:16:37.175226 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:37.175144    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chw9t_ddbaa431-1284-4b0c-8f63-585d594e0473/node-exporter/0.log"
Jul 16 22:16:37.221582 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:37.221509    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chw9t_ddbaa431-1284-4b0c-8f63-585d594e0473/kube-rbac-proxy/0.log"
Jul 16 22:16:37.268535 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:37.268511    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chw9t_ddbaa431-1284-4b0c-8f63-585d594e0473/init-textfile/0.log"
Jul 16 22:16:39.492127 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.492074    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"]
Jul 16 22:16:39.497545 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.497519    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.506670 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.506644    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"]
Jul 16 22:16:39.589529 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.589490    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c79413d5-c201-4faf-adc3-3236a974678a-sys\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.589701 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.589554    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c79413d5-c201-4faf-adc3-3236a974678a-proc\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.589701 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.589598    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2flx\" (UniqueName: \"kubernetes.io/projected/c79413d5-c201-4faf-adc3-3236a974678a-kube-api-access-t2flx\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.589701 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.589678    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c79413d5-c201-4faf-adc3-3236a974678a-podres\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.589885 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.589712    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c79413d5-c201-4faf-adc3-3236a974678a-lib-modules\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.690418 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.690376    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c79413d5-c201-4faf-adc3-3236a974678a-proc\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.690599 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.690444    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2flx\" (UniqueName: \"kubernetes.io/projected/c79413d5-c201-4faf-adc3-3236a974678a-kube-api-access-t2flx\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.690599 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.690500    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c79413d5-c201-4faf-adc3-3236a974678a-podres\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.690599 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.690536    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c79413d5-c201-4faf-adc3-3236a974678a-lib-modules\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.690599 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.690574    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c79413d5-c201-4faf-adc3-3236a974678a-sys\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.690832 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.690679    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c79413d5-c201-4faf-adc3-3236a974678a-sys\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.690832 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.690746    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c79413d5-c201-4faf-adc3-3236a974678a-proc\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.691190 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.691158    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c79413d5-c201-4faf-adc3-3236a974678a-lib-modules\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.691190 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.691182    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c79413d5-c201-4faf-adc3-3236a974678a-podres\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.700839 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.700809    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2flx\" (UniqueName: \"kubernetes.io/projected/c79413d5-c201-4faf-adc3-3236a974678a-kube-api-access-t2flx\") pod \"perf-node-gather-daemonset-d8p8b\" (UID: \"c79413d5-c201-4faf-adc3-3236a974678a\") " pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.818017 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.817932    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:39.981967 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:39.980771    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"]
Jul 16 22:16:39.983231 ip-10-0-137-85 kubenswrapper[2570]: W0716 22:16:39.983197    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc79413d5_c201_4faf_adc3_3236a974678a.slice/crio-1666b26308c4693376cc9d2f62cb325c9bc7acedb69fbc3c500cd541c0418a9a WatchSource:0}: Error finding container 1666b26308c4693376cc9d2f62cb325c9bc7acedb69fbc3c500cd541c0418a9a: Status 404 returned error can't find the container with id 1666b26308c4693376cc9d2f62cb325c9bc7acedb69fbc3c500cd541c0418a9a
Jul 16 22:16:40.730343 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:40.730300    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b" event={"ID":"c79413d5-c201-4faf-adc3-3236a974678a","Type":"ContainerStarted","Data":"5300e1e9d1889c827aadebf16e1f899890b7c552ab5397f4cc367c15ef0c9088"}
Jul 16 22:16:40.730810 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:40.730586    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b" event={"ID":"c79413d5-c201-4faf-adc3-3236a974678a","Type":"ContainerStarted","Data":"1666b26308c4693376cc9d2f62cb325c9bc7acedb69fbc3c500cd541c0418a9a"}
Jul 16 22:16:40.730946 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:40.730922    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:40.753027 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:40.752972    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b" podStartSLOduration=1.752952698 podStartE2EDuration="1.752952698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:16:40.752417078 +0000 UTC m=+1585.349957310" watchObservedRunningTime="2026-07-16 22:16:40.752952698 +0000 UTC m=+1585.350492931"
Jul 16 22:16:41.728076 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:41.728039    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6b842_f8a6451d-57f1-412b-9f81-e2903dfdff69/dns/0.log"
Jul 16 22:16:41.750040 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:41.750014    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6b842_f8a6451d-57f1-412b-9f81-e2903dfdff69/kube-rbac-proxy/0.log"
Jul 16 22:16:41.850509 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:41.850477    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t54hm_9ae773a5-b7eb-4346-a16f-186c5f0ec2a8/dns-node-resolver/0.log"
Jul 16 22:16:42.349254 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:42.349193    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7wss5_902dfcac-4523-4a6a-9338-cbe2661ad1ec/node-ca/0.log"
Jul 16 22:16:43.490686 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:43.490657    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zq96d_ea5a4ca6-5adf-47f1-a1e4-4c59f554714d/serve-healthcheck-canary/0.log"
Jul 16 22:16:44.009626 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:44.009596    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r8jdg_8559cfa0-b3f8-4719-9e05-cef5ed2a5aec/kube-rbac-proxy/0.log"
Jul 16 22:16:44.030789 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:44.030773    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r8jdg_8559cfa0-b3f8-4719-9e05-cef5ed2a5aec/exporter/0.log"
Jul 16 22:16:44.055319 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:44.055294    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r8jdg_8559cfa0-b3f8-4719-9e05-cef5ed2a5aec/extractor/0.log"
Jul 16 22:16:46.222270 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:46.222238    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-cc4rq_43a455ac-3b52-4c4b-b1e5-d02e507018cf/manager/0.log"
Jul 16 22:16:46.452057 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:46.452024    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-tbnsr_639bdce7-3c63-4d0f-81e6-b2193ad13f2b/seaweedfs/0.log"
Jul 16 22:16:46.745042 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:46.745010    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k8wws/perf-node-gather-daemonset-d8p8b"
Jul 16 22:16:52.557219 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:52.557189    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4gx4_65bfdb19-9745-4a5e-842a-05df0b73d7c0/kube-multus-additional-cni-plugins/0.log"
Jul 16 22:16:52.577458 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:52.577437    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4gx4_65bfdb19-9745-4a5e-842a-05df0b73d7c0/egress-router-binary-copy/0.log"
Jul 16 22:16:52.597701 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:52.597679    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4gx4_65bfdb19-9745-4a5e-842a-05df0b73d7c0/cni-plugins/0.log"
Jul 16 22:16:52.618337 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:52.618312    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4gx4_65bfdb19-9745-4a5e-842a-05df0b73d7c0/bond-cni-plugin/0.log"
Jul 16 22:16:52.640541 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:52.640517    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4gx4_65bfdb19-9745-4a5e-842a-05df0b73d7c0/routeoverride-cni/0.log"
Jul 16 22:16:52.660567 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:52.660539    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4gx4_65bfdb19-9745-4a5e-842a-05df0b73d7c0/whereabouts-cni-bincopy/0.log"
Jul 16 22:16:52.680728 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:52.680713    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4gx4_65bfdb19-9745-4a5e-842a-05df0b73d7c0/whereabouts-cni/0.log"
Jul 16 22:16:52.715041 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:52.715016    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-njxsf_97b07f25-0e8b-4fe6-8da3-72ca51f28f18/kube-multus/0.log"
Jul 16 22:16:52.811314 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:52.811252    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c2dgn_b2365d47-8e55-4776-b882-49eead286f8c/network-metrics-daemon/0.log"
Jul 16 22:16:52.832011 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:52.831993    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c2dgn_b2365d47-8e55-4776-b882-49eead286f8c/kube-rbac-proxy/0.log"
Jul 16 22:16:54.017347 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:54.017313    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-controller/0.log"
Jul 16 22:16:54.034997 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:54.034973    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/1.log"
Jul 16 22:16:54.035188 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:54.035007    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovn-acl-logging/0.log"
Jul 16 22:16:54.053146 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:54.053116    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/kube-rbac-proxy-node/0.log"
Jul 16 22:16:54.076193 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:54.076156    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/kube-rbac-proxy-ovn-metrics/0.log"
Jul 16 22:16:54.097762 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:54.097725    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/northd/0.log"
Jul 16 22:16:54.120060 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:54.120030    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/nbdb/0.log"
Jul 16 22:16:54.146919 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:54.146890    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/sbdb/0.log"
Jul 16 22:16:54.211298 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:54.211264    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r6hvb_93848932-9ec1-45e6-aec9-34f0a623b9ef/ovnkube-controller/0.log"
Jul 16 22:16:55.259823 ip-10-0-137-85 kubenswrapper[2570]: I0716 22:16:55.259794    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-s9mhc_d5c93b6e-7ae4-4340-86bd-c10c6b804523/network-check-target-container/0.log"