Jul 08 16:32:31.906199 ip-10-0-138-223 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 08 16:32:31.906211 ip-10-0-138-223 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 08 16:32:31.906221 ip-10-0-138-223 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 08 16:32:31.906580 ip-10-0-138-223 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 08 16:32:42.061524 ip-10-0-138-223 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 08 16:32:42.061543 ip-10-0-138-223 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot babdfd0e4a0b494eb92ec32337b1b7a9 --
Jul 08 16:34:44.354861 ip-10-0-138-223 systemd[1]: Starting Kubernetes Kubelet...
Jul 08 16:34:44.725654 ip-10-0-138-223 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:44.725654 ip-10-0-138-223 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 08 16:34:44.726083 ip-10-0-138-223 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:44.726083 ip-10-0-138-223 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 08 16:34:44.726083 ip-10-0-138-223 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:44.726591 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.726513    2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 08 16:34:44.729921 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729907    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:44.729921 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729920    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729924    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729927    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729930    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729932    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729935    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729938    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729940    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729943    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729945    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729949    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729953    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729955    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729958    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729960    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729963    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729965    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729968    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729971    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729973    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:44.729978 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729975    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729978    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729980    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729983    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729985    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729989    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729993    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729996    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.729999    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730001    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730004    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730007    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730010    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730012    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730015    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730017    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730020    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730022    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730025    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:44.730426 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730027    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730029    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730032    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730034    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730037    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730039    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730041    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730044    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730047    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730049    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730051    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730053    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730055    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730058    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730060    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730063    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730066    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730069    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730071    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730074    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:44.730883 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730076    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730079    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730081    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730083    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730086    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730088    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730090    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730093    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730095    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730097    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730100    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730102    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730105    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730107    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730110    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730113    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730115    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730117    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730120    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:44.731389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730122    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730126    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730129    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730132    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730134    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730137    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730139    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730142    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730145    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730147    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730150    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730152    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730155    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730157    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730159    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730162    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730164    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730166    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730169    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:44.731854 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730171    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730827    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730834    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730837    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730839    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730842    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730845    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730847    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730850    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730852    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730854    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730858    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730860    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730863    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730865    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730867    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730870    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730872    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730875    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730877    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:44.732299 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730879    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730882    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730885    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730887    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730890    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730893    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730895    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730898    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730900    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730902    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730905    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730908    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730910    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730913    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730915    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730917    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730919    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730922    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730924    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730927    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:44.732886 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730929    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730932    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730934    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730937    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730939    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730942    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730944    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730947    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730950    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730952    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730954    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730956    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730959    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730961    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730964    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730966    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730969    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730971    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730973    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730976    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:44.733347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730979    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730981    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730983    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730985    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730988    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730990    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730992    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730995    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730997    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.730999    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731001    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731004    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731006    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731009    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731011    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731014    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731016    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731018    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731021    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731024    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:44.733838 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731026    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731030    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731034    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731037    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731041    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731043    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731046    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731048    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731052    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731056    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731059    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731062    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731064    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731067    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731069    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731072    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731074    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731078    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731081    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:44.734308 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731084    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731406    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731411    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731414    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731416    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731419    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731422    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731425    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731427    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731430    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731432    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731435    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731438    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731440    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731442    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731445    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731447    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731450    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731453    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731455    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:44.734761 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731457    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731460    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731462    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731464    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731466    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731469    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731471    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731475    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731478    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731481    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731484    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731486    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731488    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731491    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731493    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731496    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731498    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731501    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731504    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731507    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:44.735229 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731509    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731511    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731514    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731516    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731519    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731522    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731524    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731526    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731529    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731531    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731534    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731537    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731539    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731541    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731544    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731548    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731550    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731552    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731555    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:44.735703 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731557    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731577    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731580    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731582    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731584    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731587    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731590    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731592    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731595    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731597    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731600    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731602    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731605    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731608    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731610    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731613    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731615    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731617    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731620    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731622    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:44.736170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731625    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731627    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731630    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731632    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731634    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731637    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731641    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731644    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731651    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731654    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731657    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731659    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731661    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731664    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731667    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731669    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731671    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731674    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731676    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:44.736648 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731679    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.731682    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731757    2550 flags.go:64] FLAG: --address="0.0.0.0"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731767    2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731805    2550 flags.go:64] FLAG: --anonymous-auth="true"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731810    2550 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731815    2550 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731818    2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731822    2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731826    2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731829    2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731831    2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731834    2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731837    2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731840    2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731843    2550 flags.go:64] FLAG: --cgroup-root=""
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731846    2550 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731848    2550 flags.go:64] FLAG: --client-ca-file=""
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731851    2550 flags.go:64] FLAG: --cloud-provider="external"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731854    2550 flags.go:64] FLAG: --cluster-dns="[]"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731858    2550 flags.go:64] FLAG: --cluster-domain=""
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731862    2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731865    2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731868    2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 08 16:34:44.737101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731872    2550 flags.go:64] FLAG: --container-log-max-files="5"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731878    2550 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731881    2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731884    2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731887    2550 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731889    2550 flags.go:64] FLAG: --contention-profiling="false"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731892    2550 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731895    2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731898    2550 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731900    2550 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731904    2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731907    2550 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731909    2550 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731912    2550 flags.go:64] FLAG: --enable-load-reader="false"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731915    2550 flags.go:64] FLAG: --enable-server="true"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731918    2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731922    2550 flags.go:64] FLAG: --event-burst="100"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731924    2550 flags.go:64] FLAG: --event-qps="50"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731927    2550 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731930    2550 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731932    2550 flags.go:64] FLAG: --eviction-hard=""
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731936    2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731938    2550 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731942    2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731945    2550 flags.go:64] FLAG: --eviction-soft=""
Jul 08 16:34:44.737703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731948    2550 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731950    2550 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731953    2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731955    2550 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731959    2550 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731962    2550 flags.go:64] FLAG: --fail-swap-on="true"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731965    2550 flags.go:64] FLAG: --feature-gates=""
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731968    2550 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731971    2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731974    2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731977    2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731980    2550 flags.go:64] FLAG: --healthz-port="10248"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731982    2550 flags.go:64] FLAG: --help="false"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731985    2550 flags.go:64] FLAG: --hostname-override="ip-10-0-138-223.ec2.internal"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731988    2550 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731990    2550 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731993    2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731996    2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.731999    2550 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732002    2550 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732005    2550 flags.go:64] FLAG: --image-service-endpoint=""
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732007    2550 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732010    2550 flags.go:64] FLAG: --kube-api-burst="100"
Jul 08 16:34:44.738293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732013    2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732017    2550 flags.go:64] FLAG: --kube-api-qps="50"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732019    2550 flags.go:64] FLAG: --kube-reserved=""
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732022    2550 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732024    2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732027    2550 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732029    2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732033    2550 flags.go:64] FLAG: --lock-file=""
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732035    2550 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732038    2550 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732041    2550 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732046    2550 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732048    2550 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732052    2550 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732055    2550 flags.go:64] FLAG: --logging-format="text"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732057    2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732060    2550 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732063    2550 flags.go:64] FLAG: --manifest-url=""
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732065    2550 flags.go:64] FLAG: --manifest-url-header=""
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732070    2550 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732072    2550 flags.go:64] FLAG: --max-open-files="1000000"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732076    2550 flags.go:64] FLAG: --max-pods="110"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732079    2550 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732081    2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732084    2550 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 08 16:34:44.738868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732087    2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732089    2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732092    2550 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732095    2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732101    2550 flags.go:64] FLAG: --node-status-max-images="50"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732104    2550 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732107    2550 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732109    2550 flags.go:64] FLAG: --pod-cidr=""
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732112    2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732117    2550 flags.go:64] FLAG: --pod-manifest-path=""
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732120    2550 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732123    2550 flags.go:64] FLAG: --pods-per-core="0"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732125    2550 flags.go:64] FLAG: --port="10250"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732128    2550 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732131    2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0356c88ca27df0cd4"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732135    2550 flags.go:64] FLAG: --qos-reserved=""
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732138    2550 flags.go:64] FLAG: --read-only-port="10255"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732140    2550 flags.go:64] FLAG: --register-node="true"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732143    2550 flags.go:64] FLAG: --register-with-taints=""
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732146    2550 flags.go:64] FLAG: --registry-burst="10"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732151    2550 flags.go:64] FLAG: --registry-qps="5"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732153    2550 flags.go:64] FLAG: --reserved-cpus=""
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732155    2550 flags.go:64] FLAG: --reserved-memory=""
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732159    2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732161    2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 08 16:34:44.739482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732164    2550 flags.go:64] FLAG: --rotate-certificates="false"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732167    2550 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732169    2550 flags.go:64] FLAG: --runonce="false"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732172    2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732175    2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732177    2550 flags.go:64] FLAG: --seccomp-default="false"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732180    2550 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732182    2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732185    2550 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732188    2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732190    2550 flags.go:64] FLAG: --storage-driver-password="root"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732193    2550 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732195    2550 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732198    2550 flags.go:64] FLAG: --storage-driver-user="root"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732200    2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732203    2550 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732206    2550 flags.go:64] FLAG: --system-cgroups=""
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732208    2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732213    2550 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732215    2550 flags.go:64] FLAG: --tls-cert-file=""
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732218    2550 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732222    2550 flags.go:64] FLAG: --tls-min-version=""
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732227    2550 flags.go:64] FLAG: --tls-private-key-file=""
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732229    2550 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732232    2550 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 08 16:34:44.740076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732234    2550 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732237    2550 flags.go:64] FLAG: --v="2"
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732242    2550 flags.go:64] FLAG: --version="false"
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732246    2550 flags.go:64] FLAG: --vmodule=""
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732249    2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.732252    2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732360    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732363    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732366    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732369    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732372    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732374    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732377    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732379    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732381    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732384    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732386    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732388    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732391    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732393    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732395    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:44.740688 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732398    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732401    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732403    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732407    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732411    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732414    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732416    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732419    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732422    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732425    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732428    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732430    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732432    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732435    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732438    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732440    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732442    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732445    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732447    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732450    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:44.741186 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732452    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732454    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732457    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732459    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732461    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732464    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732467    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732470    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732474    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732476    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732479    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732481    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732485    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732488    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732490    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732493    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732496    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732500    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732502    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:44.741961 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732505    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732509    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732512    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732514    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732517    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732519    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732521    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732524    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732526    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732529    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732531    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732534    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732536    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732538    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732541    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732543    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732546    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732548    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732551    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732554    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:44.742517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732556    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732572    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732575    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732577    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732580    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732582    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732585    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732587    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732590    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732592    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732595    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732597    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732600    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732604    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732606    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732608    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732611    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732613    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732615    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:44.743056 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732617    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732620    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732622    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732625    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732627    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.732629    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.733261    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.739000    2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.739014    2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.739766    2550 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.739773    2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739824    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739829    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739832    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739835    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:44.743506 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739838    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739841    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739843    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739846    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739848    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739851    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739853    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739856    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739858    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739861    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739864    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739866    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739869    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739872    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739874    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739876    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739879    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739881    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739883    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739886    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:44.743894 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739888    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739891    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739893    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739896    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739898    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739902    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739905    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739908    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739911    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739913    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739915    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739918    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739920    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739922    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739925    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739927    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739929    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739932    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739935    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739937    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:44.744366 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739939    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739942    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739944    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739947    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739951    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739954    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739957    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739960    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739963    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739965    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739967    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739971    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739975    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739978    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739980    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739982    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739985    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739988    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739992    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:44.744863 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739996    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.739999    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740001    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740004    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740006    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740009    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740012    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740015    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740017    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740019    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740022    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740024    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740027    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740029    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740032    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740035    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740037    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740039    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740042    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740044    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:44.745442 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740046    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740049    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740051    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740053    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740056    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740058    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740061    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740063    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740066    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740069    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740071    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740073    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740076    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740078    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740081    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:44.745959 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740083    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.740088    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740190    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740195    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740197    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740200    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740203    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740205    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740208    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740210    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740212    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740216    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740219    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740221    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740224    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:44.746389 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740226    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740228    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740231    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740233    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740235    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740237    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740240    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740242    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740245    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740247    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740249    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740251    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740254    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740256    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740259    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740262    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740264    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740266    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740269    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740271    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:44.746869 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740273    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740276    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740278    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740280    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740283    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740285    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740287    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740289    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740292    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740294    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740296    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740300    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740303    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740306    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740308    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740311    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740313    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740315    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740318    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:44.747674 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740320    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740322    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740324    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740327    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740329    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740331    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740334    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740336    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740338    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740341    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740343    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740346    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740348    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740350    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740353    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740355    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740357    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740359    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740362    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740364    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:44.748228 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740367    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740369    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740371    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740373    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740376    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740379    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740381    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740383    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740386    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740388    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740390    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740392    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740395    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740397    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740399    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740401    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740405    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740408    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740411    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:44.748850 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740414    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740416    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740418    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740428    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740430    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740433    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740435    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:44.740438    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.740443    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.740535    2550 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.743681    2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.745006    2550 server.go:1019] "Starting client certificate rotation"
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.745123    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:34:44.749411 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.745170    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:34:44.749790 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.747394    2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:34:44.749790 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.748389    2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:34:44.760947 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.760928    2550 log.go:25] "Validated CRI v1 runtime API"
Jul 08 16:34:44.763004 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.762990    2550 log.go:25] "Validated CRI v1 image API"
Jul 08 16:34:44.763896 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.763882    2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 08 16:34:44.768666 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.768636    2550 fs.go:135] Filesystem UUIDs: map[3a6ee558-910e-4887-a297-50a82db1543c:/dev/nvme0n1p3 6ea5e4df-b02f-4d90-bde0-22a1ce74f043:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 08 16:34:44.768706 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.768668    2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 08 16:34:44.773970 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.773954    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:34:44.774768 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.774671    2550 manager.go:216] Machine: {Timestamp:2026-07-08 16:34:44.773364743 +0000 UTC m=+0.309917219 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3182627 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2567ecf1e3e707624a56db937623b6 SystemUUID:ec2567ec-f1e3-e707-624a-56db937623b6 BootID:babdfd0e-4a0b-494e-b92e-c32337b1b7a9 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c2:55:77:b6:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c2:55:77:b6:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:f6:a8:45:a0:d5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 08 16:34:44.774768 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.774767    2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 08 16:34:44.774863 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.774841    2550 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 08 16:34:44.777667 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.777643    2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 08 16:34:44.777785 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.777670    2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-223.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 08 16:34:44.778444 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.778435    2550 topology_manager.go:138] "Creating topology manager with none policy"
Jul 08 16:34:44.778472 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.778447    2550 container_manager_linux.go:309] "Creating device plugin manager"
Jul 08 16:34:44.778472 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.778454    2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:34:44.779185 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.779174    2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:34:44.780617 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.780608    2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 08 16:34:44.783639 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.783629    2550 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:34:44.783772 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.783762    2550 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 08 16:34:44.783805 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.783797    2550 kubelet.go:486] "Attempting to sync node with API server"
Jul 08 16:34:44.783831 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.783810    2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 08 16:34:44.783831 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.783821    2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 08 16:34:44.783831 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.783829    2550 kubelet.go:398] "Adding apiserver pod source"
Jul 08 16:34:44.783915 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.783847    2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 08 16:34:44.784301 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.784281    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:34:44.784301 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.784300    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:34:44.787550 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.787476    2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 08 16:34:44.790509 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.790494    2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 08 16:34:44.790591 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.790523    2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 08 16:34:44.791628 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.791616    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 08 16:34:44.791676 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.791632    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 08 16:34:44.791676 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.791638    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 08 16:34:44.791676 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.791643    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 08 16:34:44.791676 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.791650    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 08 16:34:44.791676 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.791655    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 08 16:34:44.791676 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.791666    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 08 16:34:44.791676 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.791674    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 08 16:34:44.791855 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.791683    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 08 16:34:44.791855 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.791689    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 08 16:34:44.791855 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.791697    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 08 16:34:44.791855 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.791702    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 08 16:34:44.792399 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.792386    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 08 16:34:44.792432 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.792400    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 08 16:34:44.795150 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.795134    2550 server.go:1268] "Started kubelet"
Jul 08 16:34:44.795275 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.795223    2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 08 16:34:44.795358 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.795307    2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 08 16:34:44.795425 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.795386    2550 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 08 16:34:44.795910 ip-10-0-138-223 systemd[1]: Started Kubernetes Kubelet.
Jul 08 16:34:44.796469 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.796451    2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 08 16:34:44.796911 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.796859    2550 server.go:310] "Adding debug handlers to kubelet server"
Jul 08 16:34:44.798672 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.798654    2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-223.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 08 16:34:44.799213 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.799191    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 08 16:34:44.799213 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.799193    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 08 16:34:44.800342 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.800325    2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 08 16:34:44.801691 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.801677    2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 08 16:34:44.801742 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.801693    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 08 16:34:44.801958 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.801933    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 08 16:34:44.802411 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.802396    2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 08 16:34:44.802411 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.802399    2550 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 08 16:34:44.802537 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.802421    2550 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 08 16:34:44.802609 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.802551    2550 reconstruct.go:97] "Volume reconstruction finished"
Jul 08 16:34:44.802609 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.802578    2550 reconciler.go:29] "Reconciler: start to sync state"
Jul 08 16:34:44.802753 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.802738    2550 factory.go:55] Registering systemd factory
Jul 08 16:34:44.802829 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.802770    2550 factory.go:223] Registration of the systemd container factory successfully
Jul 08 16:34:44.802966 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.802946    2550 factory.go:153] Registering CRI-O factory
Jul 08 16:34:44.802966 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.802963    2550 factory.go:223] Registration of the crio container factory successfully
Jul 08 16:34:44.803096 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.803019    2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 08 16:34:44.803096 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.803047    2550 factory.go:103] Registering Raw factory
Jul 08 16:34:44.803096 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.803061    2550 manager.go:1196] Started watching for new ooms in manager
Jul 08 16:34:44.803397 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.802576    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-223.ec2.internal.18c05db51700bddc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-223.ec2.internal,UID:ip-10-0-138-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-223.ec2.internal,},FirstTimestamp:2026-07-08 16:34:44.795104732 +0000 UTC m=+0.331657208,LastTimestamp:2026-07-08 16:34:44.795104732 +0000 UTC m=+0.331657208,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-223.ec2.internal,}"
Jul 08 16:34:44.804279 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.804265    2550 manager.go:349] Starting recovery of all containers
Jul 08 16:34:44.809868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.809848    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d744p"
Jul 08 16:34:44.810899 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.810835    2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 08 16:34:44.811113 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.811091    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 08 16:34:44.815609 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.815590    2550 manager.go:354] Recovery completed
Jul 08 16:34:44.816950 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.816850    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d744p"
Jul 08 16:34:44.820090 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.820079    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:44.821846 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.821826    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:44.821916 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.821866    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:44.821916 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.821880    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:44.822413 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.822398    2550 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 08 16:34:44.822413 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.822411    2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 08 16:34:44.822544 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.822416    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:44.822544 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.822425    2550 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:34:44.823140 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.823127    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:44.823183 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.823152    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:44.823183 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.823161    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:44.823408 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.823398    2550 policy_none.go:49] "None policy: Start"
Jul 08 16:34:44.823440 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.823413    2550 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 08 16:34:44.823440 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.823422    2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 08 16:34:44.824159 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.824100    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-223.ec2.internal.18c05db51898cd19  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-223.ec2.internal,UID:ip-10-0-138-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-223.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-223.ec2.internal,},FirstTimestamp:2026-07-08 16:34:44.821847321 +0000 UTC m=+0.358399988,LastTimestamp:2026-07-08 16:34:44.821847321 +0000 UTC m=+0.358399988,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-223.ec2.internal,}"
Jul 08 16:34:44.824352 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.824340    2550 policy_none.go:47] "Start"
Jul 08 16:34:44.865088 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.865075    2550 manager.go:337] "Starting Device Plugin manager"
Jul 08 16:34:44.865178 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.865100    2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 08 16:34:44.865178 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.865109    2550 server.go:87] "Starting device plugin registration server"
Jul 08 16:34:44.865297 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.865283    2550 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 08 16:34:44.865341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.865294    2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 08 16:34:44.865391 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.865366    2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 08 16:34:44.865443 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.865427    2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 08 16:34:44.865443 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.865435    2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 08 16:34:44.865911 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.865892    2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 08 16:34:44.865982 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.865946    2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 08 16:34:44.966022 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.966002    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:44.967099 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.966742    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:44.967099 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.966769    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:44.967099 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.966780    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:44.967099 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.966799    2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-223.ec2.internal"
Jul 08 16:34:44.973051 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:44.973037    2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-223.ec2.internal"
Jul 08 16:34:44.973096 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.973055    2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-223.ec2.internal\": node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 08 16:34:44.986537 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:44.986489    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 08 16:34:45.020948 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.020921    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 08 16:34:45.022191 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.022176    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 08 16:34:45.022261 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.022218    2550 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 08 16:34:45.022261 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.022239    2550 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 08 16:34:45.022356 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:45.022273    2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 08 16:34:45.025631 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.025604    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:45.087622 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:45.087592    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 08 16:34:45.123008 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.122982    2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"]
Jul 08 16:34:45.123068 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.123048    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:45.123868 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.123854    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:45.123940 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.123875    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:45.123940 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.123885    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:45.125154 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.125143    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:45.125271 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.125260    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.125312 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.125284    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:45.125788 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.125774    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:45.125870 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.125798    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:45.125870 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.125804    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:45.125870 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.125813    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:45.125870 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.125822    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:45.125870 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.125844    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:45.126983 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.126966    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.127067 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.126990    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:45.127603 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.127586    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:45.127686 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.127613    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:45.127686 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.127627    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:45.139069 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:45.139046    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-223.ec2.internal\" not found" node="ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.142985 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:45.142970    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-223.ec2.internal\" not found" node="ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.188044 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:45.188027    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 08 16:34:45.204726 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.204708    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9c94162aa2f7a295900425e7a7b11cc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"9c94162aa2f7a295900425e7a7b11cc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.204772 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.204731    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c94162aa2f7a295900425e7a7b11cc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"9c94162aa2f7a295900425e7a7b11cc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.204772 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.204747    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f47587f1ae5d88e48c97159a168407ff-config\") pod \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" (UID: \"f47587f1ae5d88e48c97159a168407ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.288848 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:45.288803    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 08 16:34:45.305169 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.305150    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9c94162aa2f7a295900425e7a7b11cc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"9c94162aa2f7a295900425e7a7b11cc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.305214 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.305173    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c94162aa2f7a295900425e7a7b11cc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"9c94162aa2f7a295900425e7a7b11cc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.305214 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.305188    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f47587f1ae5d88e48c97159a168407ff-config\") pod \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" (UID: \"f47587f1ae5d88e48c97159a168407ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.305275 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.305212    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f47587f1ae5d88e48c97159a168407ff-config\") pod \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" (UID: \"f47587f1ae5d88e48c97159a168407ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.305275 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.305232    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9c94162aa2f7a295900425e7a7b11cc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"9c94162aa2f7a295900425e7a7b11cc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.305275 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.305243    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c94162aa2f7a295900425e7a7b11cc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"9c94162aa2f7a295900425e7a7b11cc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.389292 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:45.389277    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 08 16:34:45.441766 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.441745    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.446202 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.446185    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.490263 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:45.490240    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 08 16:34:45.590810 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:45.590766    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 08 16:34:45.691288 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:45.691251    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 08 16:34:45.744805 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.744778    2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 08 16:34:45.745213 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.744909    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 16:34:45.792103 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:45.792080    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 08 16:34:45.821626 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.821589    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 16:29:44 +0000 UTC" deadline="2028-02-23 08:19:09.823882847 +0000 UTC"
Jul 08 16:34:45.821626 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.821626    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14271h44m24.002260976s"
Jul 08 16:34:45.858763 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.858710    2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:45.868551 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.868530    2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:45.901905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.901878    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.913070 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.913046    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:34:45.913956 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.913941    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 08 16:34:45.926237 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.926222    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:34:45.935677 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:45.935652    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47587f1ae5d88e48c97159a168407ff.slice/crio-7b141aaa580fca12a68ee2c78635ff54ac65a79fb3621e6e496b30e6a40edbbc WatchSource:0}: Error finding container 7b141aaa580fca12a68ee2c78635ff54ac65a79fb3621e6e496b30e6a40edbbc: Status 404 returned error can't find the container with id 7b141aaa580fca12a68ee2c78635ff54ac65a79fb3621e6e496b30e6a40edbbc
Jul 08 16:34:45.940280 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:45.940262    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:34:46.025697 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.025660    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" event={"ID":"f47587f1ae5d88e48c97159a168407ff","Type":"ContainerStarted","Data":"7b141aaa580fca12a68ee2c78635ff54ac65a79fb3621e6e496b30e6a40edbbc"}
Jul 08 16:34:46.152517 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:46.152499    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c94162aa2f7a295900425e7a7b11cc8.slice/crio-47a3b8c11b3653335785457eacd0dbc9159c9edc5bb33bae02b64f0df93ccbd5 WatchSource:0}: Error finding container 47a3b8c11b3653335785457eacd0dbc9159c9edc5bb33bae02b64f0df93ccbd5: Status 404 returned error can't find the container with id 47a3b8c11b3653335785457eacd0dbc9159c9edc5bb33bae02b64f0df93ccbd5
Jul 08 16:34:46.269975 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.269951    2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:46.786086 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.786058    2550 apiserver.go:52] "Watching apiserver"
Jul 08 16:34:46.793261 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.793237    2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 08 16:34:46.795550 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.795513    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-gm9kt","kube-system/konnectivity-agent-69j4d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8","openshift-image-registry/node-ca-kxzc4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal","openshift-multus/multus-additional-cni-plugins-qp7pp","openshift-network-diagnostics/network-check-target-79p9f","openshift-ovn-kubernetes/ovnkube-node-lzzb2","kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2sszl","openshift-dns/node-resolver-4dpm5","openshift-multus/multus-x8vzr","openshift-multus/network-metrics-daemon-xphrq","openshift-network-operator/iptables-alerter-t6n9h"]
Jul 08 16:34:46.797116 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.797097    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:46.797216 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:46.797166    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:34:46.798399 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.798375    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4dpm5"
Jul 08 16:34:46.799709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.799689    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.801453 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.800820    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wjj5k\""
Jul 08 16:34:46.801453 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.801036    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 08 16:34:46.801453 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.801135    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kxzc4"
Jul 08 16:34:46.801453 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.801319    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 08 16:34:46.801939 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.801923    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 08 16:34:46.802177 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.802158    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 08 16:34:46.802259 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.802245    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 08 16:34:46.803611 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.802777    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4mwfb\""
Jul 08 16:34:46.803611 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.802994    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t6n9h"
Jul 08 16:34:46.803611 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.803040    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 08 16:34:46.803611 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.803482    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 08 16:34:46.803882 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.803737    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nrkj8\""
Jul 08 16:34:46.803882 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.803761    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 08 16:34:46.803882 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.803738    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 08 16:34:46.804519 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.804502    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:46.805828 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.805806    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:46.806084 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.806065    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:46.806197 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.806175    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zhq8c\""
Jul 08 16:34:46.806324 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.806229    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.806994 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.806978    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 08 16:34:46.809378 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.807573    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 08 16:34:46.809378 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.808462    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 08 16:34:46.809378 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.808740    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fhppv\""
Jul 08 16:34:46.809378 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.809288    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 08 16:34:46.809581 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.809508    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 08 16:34:46.809842 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.809814    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 08 16:34:46.810380 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.810358    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 08 16:34:46.810486 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.810468    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 08 16:34:46.810595 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.810576    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xgqlw\""
Jul 08 16:34:46.811193 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.811173    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.812712    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.812812    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1dcc1bed-da11-489e-801a-904a8a21583a-host\") pod \"node-ca-kxzc4\" (UID: \"1dcc1bed-da11-489e-801a-904a8a21583a\") " pod="openshift-image-registry/node-ca-kxzc4"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.812861    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzxnc\" (UniqueName: \"kubernetes.io/projected/bb3cafd2-b2e5-44b5-8f40-36e9e70b9551-kube-api-access-mzxnc\") pod \"iptables-alerter-t6n9h\" (UID: \"bb3cafd2-b2e5-44b5-8f40-36e9e70b9551\") " pod="openshift-network-operator/iptables-alerter-t6n9h"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.812902    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-registration-dir\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.812931    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjpcr\" (UniqueName: \"kubernetes.io/projected/ef171d74-d49f-4979-8d12-09ab8514e543-kube-api-access-hjpcr\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.812964    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb3cafd2-b2e5-44b5-8f40-36e9e70b9551-iptables-alerter-script\") pod \"iptables-alerter-t6n9h\" (UID: \"bb3cafd2-b2e5-44b5-8f40-36e9e70b9551\") " pod="openshift-network-operator/iptables-alerter-t6n9h"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.812997    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-etc-selinux\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813029    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-sys-fs\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813058    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb3cafd2-b2e5-44b5-8f40-36e9e70b9551-host-slash\") pod \"iptables-alerter-t6n9h\" (UID: \"bb3cafd2-b2e5-44b5-8f40-36e9e70b9551\") " pod="openshift-network-operator/iptables-alerter-t6n9h"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813084    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sgnv\" (UniqueName: \"kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv\") pod \"network-check-target-79p9f\" (UID: \"d22dbf53-7ebd-4312-a491-894b8392147d\") " pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813261    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/248e79a6-fa20-4328-a4e7-0c6c54c93cd9-hosts-file\") pod \"node-resolver-4dpm5\" (UID: \"248e79a6-fa20-4328-a4e7-0c6c54c93cd9\") " pod="openshift-dns/node-resolver-4dpm5"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813312    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813514    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248e79a6-fa20-4328-a4e7-0c6c54c93cd9-tmp-dir\") pod \"node-resolver-4dpm5\" (UID: \"248e79a6-fa20-4328-a4e7-0c6c54c93cd9\") " pod="openshift-dns/node-resolver-4dpm5"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813545    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779c9\" (UniqueName: \"kubernetes.io/projected/248e79a6-fa20-4328-a4e7-0c6c54c93cd9-kube-api-access-779c9\") pod \"node-resolver-4dpm5\" (UID: \"248e79a6-fa20-4328-a4e7-0c6c54c93cd9\") " pod="openshift-dns/node-resolver-4dpm5"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813719    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813761    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-socket-dir\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813795    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1dcc1bed-da11-489e-801a-904a8a21583a-serviceca\") pod \"node-ca-kxzc4\" (UID: \"1dcc1bed-da11-489e-801a-904a8a21583a\") " pod="openshift-image-registry/node-ca-kxzc4"
Jul 08 16:34:46.815212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813869    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7w4l\" (UniqueName: \"kubernetes.io/projected/1dcc1bed-da11-489e-801a-904a8a21583a-kube-api-access-j7w4l\") pod \"node-ca-kxzc4\" (UID: \"1dcc1bed-da11-489e-801a-904a8a21583a\") " pod="openshift-image-registry/node-ca-kxzc4"
Jul 08 16:34:46.816019 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813919    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.816019 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.813999    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-device-dir\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.816019 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.814222    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:46.816019 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:46.814297    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:34:46.816216 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.816013    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:34:46.816372 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.816356    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-69j4d"
Jul 08 16:34:46.817664 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.817329    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.818786 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.818766    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:46.818885 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:46.818859    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:34:46.819524 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.819503    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:46.820100 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.819739    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 08 16:34:46.820100 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.819966    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8jn6c\""
Jul 08 16:34:46.820100 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.820048    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.820449 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.820432    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f5krq\""
Jul 08 16:34:46.820530 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.820495    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:46.821041 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.821019    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 08 16:34:46.823820 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.823802    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 08 16:34:46.823916 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.823866    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6cd8j\""
Jul 08 16:34:46.849407 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.849387    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r48ww"
Jul 08 16:34:46.858498 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.858477    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r48ww"
Jul 08 16:34:46.903632 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.903596    2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 08 16:34:46.915137 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915113    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-sys-fs\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.915241 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915152    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb3cafd2-b2e5-44b5-8f40-36e9e70b9551-host-slash\") pod \"iptables-alerter-t6n9h\" (UID: \"bb3cafd2-b2e5-44b5-8f40-36e9e70b9551\") " pod="openshift-network-operator/iptables-alerter-t6n9h"
Jul 08 16:34:46.915241 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915183    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-cni-netd\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.915241 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915210    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-run-netns\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.915241 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915233    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/248e79a6-fa20-4328-a4e7-0c6c54c93cd9-hosts-file\") pod \"node-resolver-4dpm5\" (UID: \"248e79a6-fa20-4328-a4e7-0c6c54c93cd9\") " pod="openshift-dns/node-resolver-4dpm5"
Jul 08 16:34:46.915461 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915279    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779c9\" (UniqueName: \"kubernetes.io/projected/248e79a6-fa20-4328-a4e7-0c6c54c93cd9-kube-api-access-779c9\") pod \"node-resolver-4dpm5\" (UID: \"248e79a6-fa20-4328-a4e7-0c6c54c93cd9\") " pod="openshift-dns/node-resolver-4dpm5"
Jul 08 16:34:46.915461 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915289    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb3cafd2-b2e5-44b5-8f40-36e9e70b9551-host-slash\") pod \"iptables-alerter-t6n9h\" (UID: \"bb3cafd2-b2e5-44b5-8f40-36e9e70b9551\") " pod="openshift-network-operator/iptables-alerter-t6n9h"
Jul 08 16:34:46.915461 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915295    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/248e79a6-fa20-4328-a4e7-0c6c54c93cd9-hosts-file\") pod \"node-resolver-4dpm5\" (UID: \"248e79a6-fa20-4328-a4e7-0c6c54c93cd9\") " pod="openshift-dns/node-resolver-4dpm5"
Jul 08 16:34:46.915461 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915354    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-sys-fs\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.915461 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915385    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-run-ovn\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.915731 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915494    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-ovnkube-config\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.915731 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915532    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-sysconfig\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.915731 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915552    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-tmp\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.915731 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915598    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-multus-conf-dir\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.915731 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915616    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-etc-kubernetes\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.915731 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915633    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzxnc\" (UniqueName: \"kubernetes.io/projected/bb3cafd2-b2e5-44b5-8f40-36e9e70b9551-kube-api-access-mzxnc\") pod \"iptables-alerter-t6n9h\" (UID: \"bb3cafd2-b2e5-44b5-8f40-36e9e70b9551\") " pod="openshift-network-operator/iptables-alerter-t6n9h"
Jul 08 16:34:46.915731 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915708    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2c866b7-440e-4458-bfdf-fc786d235674-cnibin\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915740    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-systemd-units\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915764    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-node-log\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915788    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915813    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-cni-bin\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915842    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915883    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-system-cni-dir\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915906    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-var-lib-openvswitch\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915930    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-etc-openvswitch\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915957    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-hostroot\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.915985    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2c866b7-440e-4458-bfdf-fc786d235674-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916012    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7106a7f-6389-4c33-ac08-b08cde3dc850-cni-binary-copy\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916042    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgnv\" (UniqueName: \"kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv\") pod \"network-check-target-79p9f\" (UID: \"d22dbf53-7ebd-4312-a491-894b8392147d\") " pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:46.916069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916070    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-socket-dir\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916096    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7s4c\" (UniqueName: \"kubernetes.io/projected/787b3d58-fe95-4bac-a035-bd65399a7f30-kube-api-access-p7s4c\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916163    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-log-socket\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916198    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-sysctl-d\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916219    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-run\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916240    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-run-multus-certs\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916257    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1dcc1bed-da11-489e-801a-904a8a21583a-host\") pod \"node-ca-kxzc4\" (UID: \"1dcc1bed-da11-489e-801a-904a8a21583a\") " pod="openshift-image-registry/node-ca-kxzc4"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916291    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2c866b7-440e-4458-bfdf-fc786d235674-os-release\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916308    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5psrn\" (UniqueName: \"kubernetes.io/projected/d2c866b7-440e-4458-bfdf-fc786d235674-kube-api-access-5psrn\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916350    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-run-systemd\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916368    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-run-openvswitch\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916382    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-sysctl-conf\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916412    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-os-release\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916432    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-registration-dir\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916457    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbjt\" (UniqueName: \"kubernetes.io/projected/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-kube-api-access-rfbjt\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916471    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-registration-dir\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916476    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-lib-modules\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.916715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916495    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1dcc1bed-da11-489e-801a-904a8a21583a-host\") pod \"node-ca-kxzc4\" (UID: \"1dcc1bed-da11-489e-801a-904a8a21583a\") " pod="openshift-image-registry/node-ca-kxzc4"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916412    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-socket-dir\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916519    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-var-lib-kubelet\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916540    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/998f1936-ebcf-489b-86d0-e6ff22c6f840-kubelet-config\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916604    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfnmx\" (UniqueName: \"kubernetes.io/projected/b7106a7f-6389-4c33-ac08-b08cde3dc850-kube-api-access-pfnmx\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916678    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-ovnkube-script-lib\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916708    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-kubernetes\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916754    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-etc-selinux\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916858    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2c866b7-440e-4458-bfdf-fc786d235674-cni-binary-copy\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916874    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-etc-selinux\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916894    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2c866b7-440e-4458-bfdf-fc786d235674-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916926    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-kubelet\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.916965    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-env-overrides\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917017    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-multus-cni-dir\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917049    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248e79a6-fa20-4328-a4e7-0c6c54c93cd9-tmp-dir\") pod \"node-resolver-4dpm5\" (UID: \"248e79a6-fa20-4328-a4e7-0c6c54c93cd9\") " pod="openshift-dns/node-resolver-4dpm5"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917096    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2c866b7-440e-4458-bfdf-fc786d235674-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:46.917880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917133    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917202    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-ovn-node-metrics-cert\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917232    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-var-lib-cni-bin\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917275    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917299    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-device-dir\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917326    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2c866b7-440e-4458-bfdf-fc786d235674-system-cni-dir\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917353    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-systemd\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917367    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-sys\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917388    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjpcr\" (UniqueName: \"kubernetes.io/projected/ef171d74-d49f-4979-8d12-09ab8514e543-kube-api-access-hjpcr\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917393    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248e79a6-fa20-4328-a4e7-0c6c54c93cd9-tmp-dir\") pod \"node-resolver-4dpm5\" (UID: \"248e79a6-fa20-4328-a4e7-0c6c54c93cd9\") " pod="openshift-dns/node-resolver-4dpm5"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917411    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/549c7ec1-2fc1-4af7-9b87-50955730f16c-konnectivity-ca\") pod \"konnectivity-agent-69j4d\" (UID: \"549c7ec1-2fc1-4af7-9b87-50955730f16c\") " pod="kube-system/konnectivity-agent-69j4d"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917434    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/998f1936-ebcf-489b-86d0-e6ff22c6f840-dbus\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917452    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-cnibin\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917470    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-tuned\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917510    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-var-lib-cni-multus\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917526    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1dcc1bed-da11-489e-801a-904a8a21583a-serviceca\") pod \"node-ca-kxzc4\" (UID: \"1dcc1bed-da11-489e-801a-904a8a21583a\") " pod="openshift-image-registry/node-ca-kxzc4"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917540    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7w4l\" (UniqueName: \"kubernetes.io/projected/1dcc1bed-da11-489e-801a-904a8a21583a-kube-api-access-j7w4l\") pod \"node-ca-kxzc4\" (UID: \"1dcc1bed-da11-489e-801a-904a8a21583a\") " pod="openshift-image-registry/node-ca-kxzc4"
Jul 08 16:34:46.918823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917556    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917570    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-slash\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917607    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/549c7ec1-2fc1-4af7-9b87-50955730f16c-agent-certs\") pod \"konnectivity-agent-69j4d\" (UID: \"549c7ec1-2fc1-4af7-9b87-50955730f16c\") " pod="kube-system/konnectivity-agent-69j4d"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917633    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-run-netns\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917678    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-host\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917703    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-multus-socket-dir-parent\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917729    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-run-k8s-cni-cncf-io\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917753    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-var-lib-kubelet\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917774    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7106a7f-6389-4c33-ac08-b08cde3dc850-multus-daemon-config\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917799    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb3cafd2-b2e5-44b5-8f40-36e9e70b9551-iptables-alerter-script\") pod \"iptables-alerter-t6n9h\" (UID: \"bb3cafd2-b2e5-44b5-8f40-36e9e70b9551\") " pod="openshift-network-operator/iptables-alerter-t6n9h"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917824    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-modprobe-d\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.917900    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thv57\" (UniqueName: \"kubernetes.io/projected/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-kube-api-access-thv57\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.918014    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.918059    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ef171d74-d49f-4979-8d12-09ab8514e543-device-dir\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.919458    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb3cafd2-b2e5-44b5-8f40-36e9e70b9551-iptables-alerter-script\") pod \"iptables-alerter-t6n9h\" (UID: \"bb3cafd2-b2e5-44b5-8f40-36e9e70b9551\") " pod="openshift-network-operator/iptables-alerter-t6n9h"
Jul 08 16:34:46.919709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.919548    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1dcc1bed-da11-489e-801a-904a8a21583a-serviceca\") pod \"node-ca-kxzc4\" (UID: \"1dcc1bed-da11-489e-801a-904a8a21583a\") " pod="openshift-image-registry/node-ca-kxzc4"
Jul 08 16:34:46.930814 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.930238    2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 08 16:34:46.930814 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:46.930681    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:46.930814 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:46.930698    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:46.930814 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:46.930708    2550 projected.go:196] Error preparing data for projected volume kube-api-access-4sgnv for pod openshift-network-diagnostics/network-check-target-79p9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:46.930814 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:46.930792    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv podName:d22dbf53-7ebd-4312-a491-894b8392147d nodeName:}" failed. No retries permitted until 2026-07-08 16:34:47.430760268 +0000 UTC m=+2.967312749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4sgnv" (UniqueName: "kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv") pod "network-check-target-79p9f" (UID: "d22dbf53-7ebd-4312-a491-894b8392147d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:46.939636 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.939609    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-779c9\" (UniqueName: \"kubernetes.io/projected/248e79a6-fa20-4328-a4e7-0c6c54c93cd9-kube-api-access-779c9\") pod \"node-resolver-4dpm5\" (UID: \"248e79a6-fa20-4328-a4e7-0c6c54c93cd9\") " pod="openshift-dns/node-resolver-4dpm5"
Jul 08 16:34:46.940092 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.940069    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjpcr\" (UniqueName: \"kubernetes.io/projected/ef171d74-d49f-4979-8d12-09ab8514e543-kube-api-access-hjpcr\") pod \"aws-ebs-csi-driver-node-7t8d8\" (UID: \"ef171d74-d49f-4979-8d12-09ab8514e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:46.940490 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.940469    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7w4l\" (UniqueName: \"kubernetes.io/projected/1dcc1bed-da11-489e-801a-904a8a21583a-kube-api-access-j7w4l\") pod \"node-ca-kxzc4\" (UID: \"1dcc1bed-da11-489e-801a-904a8a21583a\") " pod="openshift-image-registry/node-ca-kxzc4"
Jul 08 16:34:46.962349 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:46.962284    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzxnc\" (UniqueName: \"kubernetes.io/projected/bb3cafd2-b2e5-44b5-8f40-36e9e70b9551-kube-api-access-mzxnc\") pod \"iptables-alerter-t6n9h\" (UID: \"bb3cafd2-b2e5-44b5-8f40-36e9e70b9551\") " pod="openshift-network-operator/iptables-alerter-t6n9h"
Jul 08 16:34:47.018515 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018489    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbjt\" (UniqueName: \"kubernetes.io/projected/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-kube-api-access-rfbjt\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.018646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018528    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-lib-modules\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.018646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018554    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-var-lib-kubelet\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.018731 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018695    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/998f1936-ebcf-489b-86d0-e6ff22c6f840-kubelet-config\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:47.018731 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018719    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-lib-modules\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.018810 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018735    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfnmx\" (UniqueName: \"kubernetes.io/projected/b7106a7f-6389-4c33-ac08-b08cde3dc850-kube-api-access-pfnmx\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.018810 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018788    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-var-lib-kubelet\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.018810 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018788    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-ovnkube-script-lib\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.018938 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018827    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-kubernetes\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.018938 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018867    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2c866b7-440e-4458-bfdf-fc786d235674-cni-binary-copy\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.018938 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018896    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2c866b7-440e-4458-bfdf-fc786d235674-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.018938 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018920    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-kubelet\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.019075 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018944    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-env-overrides\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.019075 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018968    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-multus-cni-dir\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.019075 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.018994    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2c866b7-440e-4458-bfdf-fc786d235674-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.019075 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019020    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.019075 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019048    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-ovn-node-metrics-cert\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.019075 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019071    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-var-lib-cni-bin\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.019295 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019101    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2c866b7-440e-4458-bfdf-fc786d235674-system-cni-dir\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.019295 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019125    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-systemd\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.019295 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019147    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-sys\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.019295 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019175    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/549c7ec1-2fc1-4af7-9b87-50955730f16c-konnectivity-ca\") pod \"konnectivity-agent-69j4d\" (UID: \"549c7ec1-2fc1-4af7-9b87-50955730f16c\") " pod="kube-system/konnectivity-agent-69j4d"
Jul 08 16:34:47.019295 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019200    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/998f1936-ebcf-489b-86d0-e6ff22c6f840-dbus\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:47.019295 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019224    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-cnibin\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.019295 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019254    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-tuned\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.019295 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019276    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-var-lib-cni-multus\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.019574 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019305    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:47.019574 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019330    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-slash\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.019574 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019368    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/549c7ec1-2fc1-4af7-9b87-50955730f16c-agent-certs\") pod \"konnectivity-agent-69j4d\" (UID: \"549c7ec1-2fc1-4af7-9b87-50955730f16c\") " pod="kube-system/konnectivity-agent-69j4d"
Jul 08 16:34:47.019574 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019394    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-run-netns\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.019574 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019408    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-ovnkube-script-lib\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.019574 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019416    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-host\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.019886 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019723    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-multus-socket-dir-parent\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.019886 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019782    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-run-k8s-cni-cncf-io\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.019886 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.019815    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-var-lib-kubelet\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.022676 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.022628    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/998f1936-ebcf-489b-86d0-e6ff22c6f840-kubelet-config\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:47.022975 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.022947    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7106a7f-6389-4c33-ac08-b08cde3dc850-multus-daemon-config\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.023091 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.022997    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-modprobe-d\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.023091 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023025    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thv57\" (UniqueName: \"kubernetes.io/projected/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-kube-api-access-thv57\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.023091 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023059    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-cni-netd\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.023091 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023088    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-run-netns\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.023316 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023120    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-run-ovn\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.023316 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023147    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-ovnkube-config\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.023316 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023178    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-sysconfig\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.023316 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023207    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-tmp\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.023316 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023236    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-multus-conf-dir\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.023316 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023266    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-etc-kubernetes\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.023316 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023294    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2c866b7-440e-4458-bfdf-fc786d235674-cnibin\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.023660 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023337    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-systemd-units\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.023660 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023367    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-node-log\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.023660 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023397    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.023660 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023428    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-cni-bin\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.023660 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023455    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:47.023660 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023486    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-system-cni-dir\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.023660 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023514    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-var-lib-openvswitch\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.023660 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023544    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-etc-openvswitch\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.023660 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023569    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-hostroot\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.023660 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023602    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2c866b7-440e-4458-bfdf-fc786d235674-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.023660 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023634    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7106a7f-6389-4c33-ac08-b08cde3dc850-cni-binary-copy\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.024082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023703    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7s4c\" (UniqueName: \"kubernetes.io/projected/787b3d58-fe95-4bac-a035-bd65399a7f30-kube-api-access-p7s4c\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:47.024082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023736    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-log-socket\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.024082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023766    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-sysctl-d\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.024082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023796    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-run\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.024082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023827    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-run-multus-certs\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.024082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023856    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2c866b7-440e-4458-bfdf-fc786d235674-os-release\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.024082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023891    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5psrn\" (UniqueName: \"kubernetes.io/projected/d2c866b7-440e-4458-bfdf-fc786d235674-kube-api-access-5psrn\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.024082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023923    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-run-systemd\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.024082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023953    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-run-openvswitch\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.024082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.023981    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-sysctl-conf\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.024082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.024013    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-os-release\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.024437 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.024173    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-os-release\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.024716    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7106a7f-6389-4c33-ac08-b08cde3dc850-multus-daemon-config\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.024848    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-modprobe-d\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.025109    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-cni-netd\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.025159    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-run-netns\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.025209    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-run-ovn\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.025704    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-ovnkube-config\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.025931    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-sysconfig\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.027273    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-hostroot\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.027331    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-multus-conf-dir\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.027367    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-etc-kubernetes\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.027406    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2c866b7-440e-4458-bfdf-fc786d235674-cnibin\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.027436    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-systemd-units\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.027469    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-node-log\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.027499    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.027534    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-cni-bin\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:47.027605    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:47.027680    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret podName:998f1936-ebcf-489b-86d0-e6ff22c6f840 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:47.527667538 +0000 UTC m=+3.064220003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret") pod "global-pull-secret-syncer-gm9kt" (UID: "998f1936-ebcf-489b-86d0-e6ff22c6f840") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:47.028470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.027734    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-system-cni-dir\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.027785    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-var-lib-openvswitch\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.027832    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-etc-openvswitch\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.027889    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-run\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.028118    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/549c7ec1-2fc1-4af7-9b87-50955730f16c-konnectivity-ca\") pod \"konnectivity-agent-69j4d\" (UID: \"549c7ec1-2fc1-4af7-9b87-50955730f16c\") " pod="kube-system/konnectivity-agent-69j4d"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.028318    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/998f1936-ebcf-489b-86d0-e6ff22c6f840-dbus\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.028393    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-cnibin\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.028516    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-log-socket\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.028660    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-sysctl-d\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.028695    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2c866b7-440e-4458-bfdf-fc786d235674-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.028856    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-kubernetes\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029039    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-kubelet\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029051    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2c866b7-440e-4458-bfdf-fc786d235674-system-cni-dir\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029065    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-systemd\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029095    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029129    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-sys\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029159    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-multus-cni-dir\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.029484 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029197    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2c866b7-440e-4458-bfdf-fc786d235674-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.030185 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029506    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-var-lib-cni-bin\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.030185 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029511    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-env-overrides\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.030185 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029622    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2c866b7-440e-4458-bfdf-fc786d235674-cni-binary-copy\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.030185 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029687    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-var-lib-cni-multus\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.030185 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029775    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-run-netns\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.030185 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029884    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-host\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.030185 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.029953    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-multus-socket-dir-parent\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.030185 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:47.030036    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:47.030185 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.030176    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-run-k8s-cni-cncf-io\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.030472 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.030220    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7106a7f-6389-4c33-ac08-b08cde3dc850-cni-binary-copy\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.030472 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.030230    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-var-lib-kubelet\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.030472 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:47.030253    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs podName:787b3d58-fe95-4bac-a035-bd65399a7f30 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:47.530102133 +0000 UTC m=+3.066654614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs") pod "network-metrics-daemon-xphrq" (UID: "787b3d58-fe95-4bac-a035-bd65399a7f30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:47.030472 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.030313    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-host-slash\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.030719 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.030695    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2c866b7-440e-4458-bfdf-fc786d235674-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.030985 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.030962    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-run-openvswitch\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.031061 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.031034    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-run-systemd\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.031115 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.031065    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2c866b7-440e-4458-bfdf-fc786d235674-os-release\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.031115 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.031091    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7106a7f-6389-4c33-ac08-b08cde3dc850-host-run-multus-certs\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.031332 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.031308    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-sysctl-conf\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.033840 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.033813    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" event={"ID":"9c94162aa2f7a295900425e7a7b11cc8","Type":"ContainerStarted","Data":"47a3b8c11b3653335785457eacd0dbc9159c9edc5bb33bae02b64f0df93ccbd5"}
Jul 08 16:34:47.034067 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.034046    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-tmp\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.034131 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.034107    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-etc-tuned\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.034194 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.034115    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-ovn-node-metrics-cert\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.036140 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.036066    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/549c7ec1-2fc1-4af7-9b87-50955730f16c-agent-certs\") pod \"konnectivity-agent-69j4d\" (UID: \"549c7ec1-2fc1-4af7-9b87-50955730f16c\") " pod="kube-system/konnectivity-agent-69j4d"
Jul 08 16:34:47.038756 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.038730    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thv57\" (UniqueName: \"kubernetes.io/projected/816f0dd7-4e9f-49d8-a0d3-fb152a6e3146-kube-api-access-thv57\") pod \"tuned-2sszl\" (UID: \"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146\") " pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.039168 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.039147    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfnmx\" (UniqueName: \"kubernetes.io/projected/b7106a7f-6389-4c33-ac08-b08cde3dc850-kube-api-access-pfnmx\") pod \"multus-x8vzr\" (UID: \"b7106a7f-6389-4c33-ac08-b08cde3dc850\") " pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.039431 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.039410    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbjt\" (UniqueName: \"kubernetes.io/projected/2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3-kube-api-access-rfbjt\") pod \"ovnkube-node-lzzb2\" (UID: \"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.041487 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.041464    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7s4c\" (UniqueName: \"kubernetes.io/projected/787b3d58-fe95-4bac-a035-bd65399a7f30-kube-api-access-p7s4c\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:47.042615 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.042559    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:47.044770 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.044738    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5psrn\" (UniqueName: \"kubernetes.io/projected/d2c866b7-440e-4458-bfdf-fc786d235674-kube-api-access-5psrn\") pod \"multus-additional-cni-plugins-qp7pp\" (UID: \"d2c866b7-440e-4458-bfdf-fc786d235674\") " pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.111550 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.111523    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4dpm5"
Jul 08 16:34:47.121872 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.121849    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8"
Jul 08 16:34:47.137803 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.137779    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kxzc4"
Jul 08 16:34:47.146067 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.146039    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t6n9h"
Jul 08 16:34:47.156474 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.156392    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qp7pp"
Jul 08 16:34:47.166650 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.166610    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:34:47.174562 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.174541    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-69j4d"
Jul 08 16:34:47.181711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.181693    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2sszl"
Jul 08 16:34:47.188384 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.188361    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x8vzr"
Jul 08 16:34:47.528079 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.528052    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgnv\" (UniqueName: \"kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv\") pod \"network-check-target-79p9f\" (UID: \"d22dbf53-7ebd-4312-a491-894b8392147d\") " pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:47.528254 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.528133    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:47.528254 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:47.528240    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:47.528372 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:47.528266    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:47.528372 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:47.528288    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:47.528372 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:47.528302    2550 projected.go:196] Error preparing data for projected volume kube-api-access-4sgnv for pod openshift-network-diagnostics/network-check-target-79p9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:47.528372 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:47.528304    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret podName:998f1936-ebcf-489b-86d0-e6ff22c6f840 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:48.528281954 +0000 UTC m=+4.064834421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret") pod "global-pull-secret-syncer-gm9kt" (UID: "998f1936-ebcf-489b-86d0-e6ff22c6f840") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:47.528372 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:47.528371    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv podName:d22dbf53-7ebd-4312-a491-894b8392147d nodeName:}" failed. No retries permitted until 2026-07-08 16:34:48.528352778 +0000 UTC m=+4.064905248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4sgnv" (UniqueName: "kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv") pod "network-check-target-79p9f" (UID: "d22dbf53-7ebd-4312-a491-894b8392147d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:47.628751 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.628722    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:47.628926 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:47.628878    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:47.628986 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:47.628944    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs podName:787b3d58-fe95-4bac-a035-bd65399a7f30 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:48.628924019 +0000 UTC m=+4.165476488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs") pod "network-metrics-daemon-xphrq" (UID: "787b3d58-fe95-4bac-a035-bd65399a7f30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:47.861220 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.861142    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:29:46 +0000 UTC" deadline="2028-03-19 01:42:41.901617446 +0000 UTC"
Jul 08 16:34:47.861220 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:47.861170    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14865h7m54.04045129s"
Jul 08 16:34:48.105484 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:48.105455    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549c7ec1_2fc1_4af7_9b87_50955730f16c.slice/crio-a0bcc6189e6c20fbb4665a85a6c4e2f6ea51a34c18bcdaad9dda7d630033ca3f WatchSource:0}: Error finding container a0bcc6189e6c20fbb4665a85a6c4e2f6ea51a34c18bcdaad9dda7d630033ca3f: Status 404 returned error can't find the container with id a0bcc6189e6c20fbb4665a85a6c4e2f6ea51a34c18bcdaad9dda7d630033ca3f
Jul 08 16:34:48.107829 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:48.107640    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef171d74_d49f_4979_8d12_09ab8514e543.slice/crio-c6f77e3ce60ed3bed833a76c0df2805a66733beba36d83e13b1b6f9c7fe7e1fe WatchSource:0}: Error finding container c6f77e3ce60ed3bed833a76c0df2805a66733beba36d83e13b1b6f9c7fe7e1fe: Status 404 returned error can't find the container with id c6f77e3ce60ed3bed833a76c0df2805a66733beba36d83e13b1b6f9c7fe7e1fe
Jul 08 16:34:48.117468 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:34:48.117389    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816f0dd7_4e9f_49d8_a0d3_fb152a6e3146.slice/crio-0ce5009331b26a29099437f25cddffb385580ae3771c255c72d27e8428d4d8af WatchSource:0}: Error finding container 0ce5009331b26a29099437f25cddffb385580ae3771c255c72d27e8428d4d8af: Status 404 returned error can't find the container with id 0ce5009331b26a29099437f25cddffb385580ae3771c255c72d27e8428d4d8af
Jul 08 16:34:48.535888 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:48.535856    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:48.535992 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:48.535903    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgnv\" (UniqueName: \"kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv\") pod \"network-check-target-79p9f\" (UID: \"d22dbf53-7ebd-4312-a491-894b8392147d\") " pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:48.536034 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:48.535992    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:48.536034 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:48.536027    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:48.536104 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:48.536043    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:48.536104 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:48.536049    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret podName:998f1936-ebcf-489b-86d0-e6ff22c6f840 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:50.536035476 +0000 UTC m=+6.072587939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret") pod "global-pull-secret-syncer-gm9kt" (UID: "998f1936-ebcf-489b-86d0-e6ff22c6f840") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:48.536104 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:48.536055    2550 projected.go:196] Error preparing data for projected volume kube-api-access-4sgnv for pod openshift-network-diagnostics/network-check-target-79p9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:48.536104 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:48.536098    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv podName:d22dbf53-7ebd-4312-a491-894b8392147d nodeName:}" failed. No retries permitted until 2026-07-08 16:34:50.53608425 +0000 UTC m=+6.072636731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4sgnv" (UniqueName: "kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv") pod "network-check-target-79p9f" (UID: "d22dbf53-7ebd-4312-a491-894b8392147d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:48.636944 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:48.636370    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:48.636944 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:48.636543    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:48.636944 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:48.636639    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs podName:787b3d58-fe95-4bac-a035-bd65399a7f30 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:50.636619097 +0000 UTC m=+6.173171565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs") pod "network-metrics-daemon-xphrq" (UID: "787b3d58-fe95-4bac-a035-bd65399a7f30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:48.861658 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:48.861398    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:29:46 +0000 UTC" deadline="2028-04-11 04:05:31.190786726 +0000 UTC"
Jul 08 16:34:48.861658 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:48.861452    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15419h30m42.329338252s"
Jul 08 16:34:49.025522 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.025449    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:49.025709 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:49.025586    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:34:49.025960 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.025939    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:49.026046 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:49.026029    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:34:49.026105 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.026089    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:49.026181 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:49.026164    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:34:49.048887 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.048855    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" event={"ID":"f47587f1ae5d88e48c97159a168407ff","Type":"ContainerStarted","Data":"19d46dd771cb0b420df3b57cd049aa190288f81e9f00dfbb37a7c0093d7ac948"}
Jul 08 16:34:49.061938 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.061911    2550 generic.go:356] "Generic (PLEG): container finished" podID="9c94162aa2f7a295900425e7a7b11cc8" containerID="c302440c2f0523bb518570b60ebf20761483109f4dfd9e7f24e09cf9eaf8fcc9" exitCode=0
Jul 08 16:34:49.062041 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.061997    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" event={"ID":"9c94162aa2f7a295900425e7a7b11cc8","Type":"ContainerDied","Data":"c302440c2f0523bb518570b60ebf20761483109f4dfd9e7f24e09cf9eaf8fcc9"}
Jul 08 16:34:49.072584 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.067894    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2sszl" event={"ID":"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146","Type":"ContainerStarted","Data":"0ce5009331b26a29099437f25cddffb385580ae3771c255c72d27e8428d4d8af"}
Jul 08 16:34:49.072584 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.070359    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2" event={"ID":"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3","Type":"ContainerStarted","Data":"683f25b1f53cbffeb04320513b18a65375952611c02da31405cfae12d7a18b46"}
Jul 08 16:34:49.072790 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.072691    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t6n9h" event={"ID":"bb3cafd2-b2e5-44b5-8f40-36e9e70b9551","Type":"ContainerStarted","Data":"626994b6d3945eb0fa619361588db3b33e717477f940b2c928c4e9197d780509"}
Jul 08 16:34:49.077352 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.077327    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x8vzr" event={"ID":"b7106a7f-6389-4c33-ac08-b08cde3dc850","Type":"ContainerStarted","Data":"1b3dcf323211a9c4a61aecbe18f5bfc74fa0108992f1e91d0ea5f48c0178073b"}
Jul 08 16:34:49.079907 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.079861    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" podStartSLOduration=3.079844821 podStartE2EDuration="3.079844821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:49.064700374 +0000 UTC m=+4.601252860" watchObservedRunningTime="2026-07-08 16:34:49.079844821 +0000 UTC m=+4.616397306"
Jul 08 16:34:49.085019 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.084992    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp7pp" event={"ID":"d2c866b7-440e-4458-bfdf-fc786d235674","Type":"ContainerStarted","Data":"9ccd7d88b281091bad986bef4127f859cc2db4fcb659c577985c137d6446be63"}
Jul 08 16:34:49.090075 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.090050    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4dpm5" event={"ID":"248e79a6-fa20-4328-a4e7-0c6c54c93cd9","Type":"ContainerStarted","Data":"ac495b6ad2e076bdf580cfa6d28f6308ee26304a203e2d385d60bd39b5de7e87"}
Jul 08 16:34:49.108212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.108187    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kxzc4" event={"ID":"1dcc1bed-da11-489e-801a-904a8a21583a","Type":"ContainerStarted","Data":"3f36ec7b53ecebb1a2e722330ebefc80c44ff8746144d3a69f484cabaecbc933"}
Jul 08 16:34:49.117296 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.117221    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8" event={"ID":"ef171d74-d49f-4979-8d12-09ab8514e543","Type":"ContainerStarted","Data":"c6f77e3ce60ed3bed833a76c0df2805a66733beba36d83e13b1b6f9c7fe7e1fe"}
Jul 08 16:34:49.132890 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:49.132868    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-69j4d" event={"ID":"549c7ec1-2fc1-4af7-9b87-50955730f16c","Type":"ContainerStarted","Data":"a0bcc6189e6c20fbb4665a85a6c4e2f6ea51a34c18bcdaad9dda7d630033ca3f"}
Jul 08 16:34:50.157646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:50.157613    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" event={"ID":"9c94162aa2f7a295900425e7a7b11cc8","Type":"ContainerStarted","Data":"af0207c9cd822ca08eb367886e4ad1662c1761a1baacf701de0ed2096b62ecdd"}
Jul 08 16:34:50.554334 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:50.554255    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:50.554334 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:50.554308    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgnv\" (UniqueName: \"kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv\") pod \"network-check-target-79p9f\" (UID: \"d22dbf53-7ebd-4312-a491-894b8392147d\") " pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:50.554546 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:50.554385    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:50.554546 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:50.554438    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:50.554546 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:50.554456    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret podName:998f1936-ebcf-489b-86d0-e6ff22c6f840 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:54.554438309 +0000 UTC m=+10.090990776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret") pod "global-pull-secret-syncer-gm9kt" (UID: "998f1936-ebcf-489b-86d0-e6ff22c6f840") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:50.554546 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:50.554458    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:50.554546 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:50.554474    2550 projected.go:196] Error preparing data for projected volume kube-api-access-4sgnv for pod openshift-network-diagnostics/network-check-target-79p9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:50.554546 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:50.554521    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv podName:d22dbf53-7ebd-4312-a491-894b8392147d nodeName:}" failed. No retries permitted until 2026-07-08 16:34:54.554506716 +0000 UTC m=+10.091059179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4sgnv" (UniqueName: "kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv") pod "network-check-target-79p9f" (UID: "d22dbf53-7ebd-4312-a491-894b8392147d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:50.655109 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:50.655075    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:50.655293 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:50.655263    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:50.655346 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:50.655327    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs podName:787b3d58-fe95-4bac-a035-bd65399a7f30 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:54.655310161 +0000 UTC m=+10.191862643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs") pod "network-metrics-daemon-xphrq" (UID: "787b3d58-fe95-4bac-a035-bd65399a7f30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:51.027062 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:51.026248    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:51.027062 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:51.026388    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:34:51.027062 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:51.026775    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:51.027062 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:51.026867    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:34:51.027062 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:51.026921    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:51.027062 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:51.026985    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:34:53.024176 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:53.024107    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:53.024595 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:53.024208    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:34:53.025524 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:53.025498    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:53.025730 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:53.025716    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:53.025843 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:53.025708    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:34:53.025976 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:53.025839    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:34:54.587813 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:54.587774    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:54.588158 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:54.587821    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgnv\" (UniqueName: \"kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv\") pod \"network-check-target-79p9f\" (UID: \"d22dbf53-7ebd-4312-a491-894b8392147d\") " pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:54.588158 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:54.587953    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:54.588158 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:54.587973    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:54.588158 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:54.587986    2550 projected.go:196] Error preparing data for projected volume kube-api-access-4sgnv for pod openshift-network-diagnostics/network-check-target-79p9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:54.588158 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:54.588043    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv podName:d22dbf53-7ebd-4312-a491-894b8392147d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:02.588026209 +0000 UTC m=+18.124578684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4sgnv" (UniqueName: "kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv") pod "network-check-target-79p9f" (UID: "d22dbf53-7ebd-4312-a491-894b8392147d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:54.588437 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:54.588411    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:54.588554 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:54.588468    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret podName:998f1936-ebcf-489b-86d0-e6ff22c6f840 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:02.588454207 +0000 UTC m=+18.125006685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret") pod "global-pull-secret-syncer-gm9kt" (UID: "998f1936-ebcf-489b-86d0-e6ff22c6f840") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:54.689119 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:54.689089    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:54.689278 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:54.689265    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:54.689330 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:54.689321    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs podName:787b3d58-fe95-4bac-a035-bd65399a7f30 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:02.689303882 +0000 UTC m=+18.225856349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs") pod "network-metrics-daemon-xphrq" (UID: "787b3d58-fe95-4bac-a035-bd65399a7f30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:55.024725 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:55.024648    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:55.024869 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:55.024777    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:34:55.025121 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:55.025101    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:55.025201 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:55.025177    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:34:55.026401 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:55.026279    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:55.026401 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:55.026370    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:34:57.023014 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:57.022977    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:57.023014 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:57.023021    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:57.023507 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:57.022977    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:57.023507 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:57.023095    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:34:57.023507 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:57.023166    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:34:57.023507 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:57.023228    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:34:59.022653 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:59.022542    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:34:59.022653 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:59.022576    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:34:59.022653 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:34:59.022597    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:34:59.023124 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:59.022688    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:34:59.023124 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:59.023063    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:34:59.023205 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:34:59.023130    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:35:01.023535 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:01.023070    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:01.023535 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:01.023070    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:01.023535 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:01.023198    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:01.023535 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:01.023210    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:35:01.023535 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:01.023322    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:35:01.023535 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:01.023420    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:35:02.653661 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:02.653633    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:02.653661 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:02.653671    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgnv\" (UniqueName: \"kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv\") pod \"network-check-target-79p9f\" (UID: \"d22dbf53-7ebd-4312-a491-894b8392147d\") " pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:02.654180 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:02.653813    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:35:02.654180 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:02.653890    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret podName:998f1936-ebcf-489b-86d0-e6ff22c6f840 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:18.653869619 +0000 UTC m=+34.190422102 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret") pod "global-pull-secret-syncer-gm9kt" (UID: "998f1936-ebcf-489b-86d0-e6ff22c6f840") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:35:02.654180 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:02.653816    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:35:02.654180 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:02.653932    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:35:02.654180 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:02.653946    2550 projected.go:196] Error preparing data for projected volume kube-api-access-4sgnv for pod openshift-network-diagnostics/network-check-target-79p9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:35:02.654180 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:02.654011    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv podName:d22dbf53-7ebd-4312-a491-894b8392147d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:18.653978464 +0000 UTC m=+34.190530940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4sgnv" (UniqueName: "kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv") pod "network-check-target-79p9f" (UID: "d22dbf53-7ebd-4312-a491-894b8392147d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:35:02.754952 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:02.754921    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:02.755077 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:02.755046    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:35:02.755121 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:02.755094    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs podName:787b3d58-fe95-4bac-a035-bd65399a7f30 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:18.75508194 +0000 UTC m=+34.291634403 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs") pod "network-metrics-daemon-xphrq" (UID: "787b3d58-fe95-4bac-a035-bd65399a7f30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:35:03.022919 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:03.022861    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:03.022919 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:03.022891    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:03.023055 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:03.022869    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:03.023055 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:03.022953    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:35:03.023055 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:03.023015    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:35:03.023140 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:03.023087    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:35:05.024249 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:05.024212    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:05.024249 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:05.024242    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:05.024770 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:05.024321    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:35:05.024770 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:05.024415    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:05.024770 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:05.024415    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:35:05.024770 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:05.024484    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:35:07.023057 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.022896    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:07.023746 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.022930    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:07.023746 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:07.023156    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:35:07.023746 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:07.023265    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:35:07.023746 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.022997    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:07.023746 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:07.023379    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:35:07.195101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.194917    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2sszl" event={"ID":"816f0dd7-4e9f-49d8-a0d3-fb152a6e3146","Type":"ContainerStarted","Data":"48c1054d8cfa825b5e6f8ff6689e7f808bca92e369758e36f7d7487831879889"}
Jul 08 16:35:07.196795 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.196776    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2" event={"ID":"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3","Type":"ContainerStarted","Data":"4e0201180a73a7cfa1ea3afd730e70b9d3aa0186de2887cbccacd41924f26bcc"}
Jul 08 16:35:07.196878 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.196846    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2" event={"ID":"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3","Type":"ContainerStarted","Data":"eb7a2de722294f5c824031d5d3ff48a9e3ae324f59f9c760d95d90ab68aba08c"}
Jul 08 16:35:07.196925 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.196893    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2" event={"ID":"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3","Type":"ContainerStarted","Data":"80c5cfaed6a31e8b01371735996ba20e44424cab9a583065606b709ed1c6c9c4"}
Jul 08 16:35:07.198097 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.198063    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x8vzr" event={"ID":"b7106a7f-6389-4c33-ac08-b08cde3dc850","Type":"ContainerStarted","Data":"0bbc08282abf56c7790e8852a686cb96e2609e22dbaf29452efb005c768772af"}
Jul 08 16:35:07.199542 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.199523    2550 generic.go:356] "Generic (PLEG): container finished" podID="d2c866b7-440e-4458-bfdf-fc786d235674" containerID="c59f691175c34bc30150ee139227c63c5fdcb12e9a91ec85849a2eee7c0b0c9b" exitCode=0
Jul 08 16:35:07.199646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.199596    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp7pp" event={"ID":"d2c866b7-440e-4458-bfdf-fc786d235674","Type":"ContainerDied","Data":"c59f691175c34bc30150ee139227c63c5fdcb12e9a91ec85849a2eee7c0b0c9b"}
Jul 08 16:35:07.200945 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.200928    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4dpm5" event={"ID":"248e79a6-fa20-4328-a4e7-0c6c54c93cd9","Type":"ContainerStarted","Data":"eaf301ff38c80de080c90395828bf1dde3ee8e0e34120c0505ec8eccdc73b308"}
Jul 08 16:35:07.202264 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.202237    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kxzc4" event={"ID":"1dcc1bed-da11-489e-801a-904a8a21583a","Type":"ContainerStarted","Data":"49899fbb68e87914362d4599305569ba81cee8a8c79b91ddf2cb8bf38399d7b4"}
Jul 08 16:35:07.203578 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.203544    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8" event={"ID":"ef171d74-d49f-4979-8d12-09ab8514e543","Type":"ContainerStarted","Data":"81009dffc2c5d053dafac8aa23a76e0bf18e523fc217a9abff19d4ac199a6f5c"}
Jul 08 16:35:07.204902 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.204884    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-69j4d" event={"ID":"549c7ec1-2fc1-4af7-9b87-50955730f16c","Type":"ContainerStarted","Data":"40ea90e9e164598e186616650694ec9d57120f42e5a09512881bddf1e1937a50"}
Jul 08 16:35:07.211343 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.211178    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" podStartSLOduration=21.211167587 podStartE2EDuration="21.211167587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:50.175243564 +0000 UTC m=+5.711796050" watchObservedRunningTime="2026-07-08 16:35:07.211167587 +0000 UTC m=+22.747720072"
Jul 08 16:35:07.214663 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.212320    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2sszl" podStartSLOduration=3.677397863 podStartE2EDuration="22.212238862s" totalImagesPullingTime="18.534840999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:07.211853029 +0000 UTC m=+22.748405509" watchObservedRunningTime="2026-07-08 16:35:07.212238862 +0000 UTC m=+22.748791347"
Jul 08 16:35:07.249645 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.249601    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4dpm5" podStartSLOduration=3.710645508 podStartE2EDuration="22.249585363s" totalImagesPullingTime="18.538939855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:07.249242903 +0000 UTC m=+22.785795388" watchObservedRunningTime="2026-07-08 16:35:07.249585363 +0000 UTC m=+22.786137848"
Jul 08 16:35:07.263062 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.263029    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kxzc4" podStartSLOduration=3.718960796 podStartE2EDuration="22.263018248s" totalImagesPullingTime="18.544057452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:07.262749535 +0000 UTC m=+22.799302019" watchObservedRunningTime="2026-07-08 16:35:07.263018248 +0000 UTC m=+22.799570733"
Jul 08 16:35:07.292402 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.292359    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x8vzr" podStartSLOduration=3.692292182 podStartE2EDuration="22.292348497s" totalImagesPullingTime="18.600056315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:07.292252835 +0000 UTC m=+22.828805321" watchObservedRunningTime="2026-07-08 16:35:07.292348497 +0000 UTC m=+22.828900981"
Jul 08 16:35:07.292643 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:07.292617    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-69j4d" podStartSLOduration=11.780124634 podStartE2EDuration="22.292608144s" totalImagesPullingTime="10.51248351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:07.275955627 +0000 UTC m=+22.812508115" watchObservedRunningTime="2026-07-08 16:35:07.292608144 +0000 UTC m=+22.829160669"
Jul 08 16:35:08.078054 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:08.078031    2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 08 16:35:08.209540 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:08.209473    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8" event={"ID":"ef171d74-d49f-4979-8d12-09ab8514e543","Type":"ContainerStarted","Data":"b8584d7351ae5ea729f431a5b3552c1ca759851ed9ec041adff3d3c6d0493feb"}
Jul 08 16:35:08.212823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:08.212794    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2" event={"ID":"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3","Type":"ContainerStarted","Data":"bc5113ee8915b6d2832244d12517700646f6cbd021fc45441ad02987f92780f2"}
Jul 08 16:35:08.212927 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:08.212865    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2" event={"ID":"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3","Type":"ContainerStarted","Data":"8957287b5fb5cc47550bb3bee3cfcc6ae0ff73e62723d7513ca1429541850f10"}
Jul 08 16:35:08.212927 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:08.212915    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2" event={"ID":"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3","Type":"ContainerStarted","Data":"1b274526ca3479bfd6c53f9bd8661b387230c9e829f26d9b1b3a790273acde39"}
Jul 08 16:35:08.214361 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:08.214330    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t6n9h" event={"ID":"bb3cafd2-b2e5-44b5-8f40-36e9e70b9551","Type":"ContainerStarted","Data":"f15a6b57c0242eb58d08607306c0be6dc90f5d69ad41daa2ed0ead04bd7d1aa4"}
Jul 08 16:35:08.228947 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:08.228897    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t6n9h" podStartSLOduration=4.695136624 podStartE2EDuration="23.228885611s" totalImagesPullingTime="18.533748987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:08.228501859 +0000 UTC m=+23.765054344" watchObservedRunningTime="2026-07-08 16:35:08.228885611 +0000 UTC m=+23.765438095"
Jul 08 16:35:08.879964 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:08.879870    2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T16:35:08.078050518Z","UUID":"32c6c5d5-9729-4821-9324-6ee30227d181","Handler":null,"Name":"","Endpoint":""}
Jul 08 16:35:08.881228 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:08.881205    2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 08 16:35:08.881347 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:08.881249    2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 08 16:35:09.023167 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:09.023137    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:09.023346 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:09.023137    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:09.023346 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:09.023259    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:35:09.023473 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:09.023357    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:35:09.023473 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:09.023138    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:09.023667 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:09.023528    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:35:09.356419 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:09.356391    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-69j4d"
Jul 08 16:35:10.222648 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:10.222613    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2" event={"ID":"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3","Type":"ContainerStarted","Data":"3219d1e3d7df55f29bf6f87d01a7f4495e5b93e465a0ba08230edbe07ca4bc41"}
Jul 08 16:35:10.224841 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:10.224808    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8" event={"ID":"ef171d74-d49f-4979-8d12-09ab8514e543","Type":"ContainerStarted","Data":"58300e6a0ba49f4023f42e01ef3ee653da6c9d5e65dfe48006eaddeb450d83b4"}
Jul 08 16:35:10.246871 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:10.246832    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7t8d8" podStartSLOduration=4.566024563 podStartE2EDuration="25.246819815s" totalImagesPullingTime="20.680795252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:10.246453348 +0000 UTC m=+25.783005835" watchObservedRunningTime="2026-07-08 16:35:10.246819815 +0000 UTC m=+25.783372300"
Jul 08 16:35:11.023134 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:11.023105    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:11.023134 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:11.023118    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:11.023683 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:11.023105    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:11.023683 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:11.023240    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:35:11.023683 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:11.023313    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:35:11.023683 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:11.023376    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:35:11.719130 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:11.718871    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-69j4d"
Jul 08 16:35:11.719660 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:11.719408    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-69j4d"
Jul 08 16:35:12.231805 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:12.231778    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2" event={"ID":"2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3","Type":"ContainerStarted","Data":"504e94ea25502233641cf5d44f14ff63b6be680774e2e739c9239e75d1cea84f"}
Jul 08 16:35:12.232482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:12.232014    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:35:12.233903 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:12.233880    2550 generic.go:356] "Generic (PLEG): container finished" podID="d2c866b7-440e-4458-bfdf-fc786d235674" containerID="b82a00b60bf08869c4cdc26993570a744de58f7ff9ab3df732ed87971c8a06b8" exitCode=0
Jul 08 16:35:12.233996 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:12.233951    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp7pp" event={"ID":"d2c866b7-440e-4458-bfdf-fc786d235674","Type":"ContainerDied","Data":"b82a00b60bf08869c4cdc26993570a744de58f7ff9ab3df732ed87971c8a06b8"}
Jul 08 16:35:12.234746 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:12.234647    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-69j4d"
Jul 08 16:35:12.247242 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:12.247226    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:35:12.258679 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:12.258645    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2" podStartSLOduration=8.655382353 podStartE2EDuration="27.258632836s" totalImagesPullingTime="18.603250483s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:12.258600066 +0000 UTC m=+27.795152543" watchObservedRunningTime="2026-07-08 16:35:12.258632836 +0000 UTC m=+27.795185303"
Jul 08 16:35:13.022958 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:13.022757    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:13.023147 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:13.022813    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:13.023147 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:13.023044    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:35:13.023147 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:13.023093    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:35:13.023147 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:13.022858    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:13.023340 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:13.023171    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:35:13.238921 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:13.238251    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:35:13.238921 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:13.238460    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:35:13.253732 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:13.253701    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:35:13.606244 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:13.606219    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gm9kt"]
Jul 08 16:35:13.606379 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:13.606327    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:13.606434 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:13.606412    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:35:13.608528 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:13.608506    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-79p9f"]
Jul 08 16:35:13.608650 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:13.608619    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:13.608714 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:13.608700    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:35:13.609314 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:13.609294    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xphrq"]
Jul 08 16:35:13.609405 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:13.609392    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:13.609503 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:13.609487    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:35:14.239932 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:14.239895    2550 generic.go:356] "Generic (PLEG): container finished" podID="d2c866b7-440e-4458-bfdf-fc786d235674" containerID="154fba7b6d5eae4b898edb78a0ae0bc509c34c77ef60b1b156090960283eaea7" exitCode=0
Jul 08 16:35:14.240476 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:14.239975    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp7pp" event={"ID":"d2c866b7-440e-4458-bfdf-fc786d235674","Type":"ContainerDied","Data":"154fba7b6d5eae4b898edb78a0ae0bc509c34c77ef60b1b156090960283eaea7"}
Jul 08 16:35:15.026401 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:15.026381    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:15.026401 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:15.026392    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:15.026604 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:15.026481    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:35:15.026704 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:15.026681    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:35:16.022910 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:16.022759    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:16.023206 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:16.023001    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:35:16.246834 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:16.246808    2550 generic.go:356] "Generic (PLEG): container finished" podID="d2c866b7-440e-4458-bfdf-fc786d235674" containerID="11513a6049e7c2c0a1d6a67853b1d94d9fa46c617b6b3d8247178f13c7f21572" exitCode=0
Jul 08 16:35:16.246951 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:16.246849    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp7pp" event={"ID":"d2c866b7-440e-4458-bfdf-fc786d235674","Type":"ContainerDied","Data":"11513a6049e7c2c0a1d6a67853b1d94d9fa46c617b6b3d8247178f13c7f21572"}
Jul 08 16:35:17.022649 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:17.022623    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:17.022804 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:17.022716    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:35:17.022804 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:17.022788    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:17.022880 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:17.022865    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:35:18.022507 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:18.022429    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:18.022943 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:18.022539    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xphrq" podUID="787b3d58-fe95-4bac-a035-bd65399a7f30"
Jul 08 16:35:18.684388 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:18.684341    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgnv\" (UniqueName: \"kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv\") pod \"network-check-target-79p9f\" (UID: \"d22dbf53-7ebd-4312-a491-894b8392147d\") " pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:18.684551 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:18.684411    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:18.684551 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:18.684496    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:35:18.684551 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:18.684505    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:35:18.684551 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:18.684526    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:35:18.684551 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:18.684535    2550 projected.go:196] Error preparing data for projected volume kube-api-access-4sgnv for pod openshift-network-diagnostics/network-check-target-79p9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:35:18.684551 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:18.684548    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret podName:998f1936-ebcf-489b-86d0-e6ff22c6f840 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:50.684535944 +0000 UTC m=+66.221088408 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret") pod "global-pull-secret-syncer-gm9kt" (UID: "998f1936-ebcf-489b-86d0-e6ff22c6f840") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:35:18.684756 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:18.684594    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv podName:d22dbf53-7ebd-4312-a491-894b8392147d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:50.684578727 +0000 UTC m=+66.221131204 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4sgnv" (UniqueName: "kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv") pod "network-check-target-79p9f" (UID: "d22dbf53-7ebd-4312-a491-894b8392147d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:35:18.785783 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:18.785753    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:18.785923 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:18.785871    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:35:18.785990 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:18.785940    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs podName:787b3d58-fe95-4bac-a035-bd65399a7f30 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:50.785918999 +0000 UTC m=+66.322471464 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs") pod "network-metrics-daemon-xphrq" (UID: "787b3d58-fe95-4bac-a035-bd65399a7f30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:35:19.023144 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.023065    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:19.023722 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:19.023195    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm9kt" podUID="998f1936-ebcf-489b-86d0-e6ff22c6f840"
Jul 08 16:35:19.023722 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.023256    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:19.023722 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:19.023354    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79p9f" podUID="d22dbf53-7ebd-4312-a491-894b8392147d"
Jul 08 16:35:19.305165 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.305078    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeReady"
Jul 08 16:35:19.305315 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.305244    2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 08 16:35:19.343720 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.343691    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65d7964d55-kbtmv"]
Jul 08 16:35:19.367443 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.367409    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rnngg"]
Jul 08 16:35:19.367623 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.367414    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.370403 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.370381    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tf9mt\""
Jul 08 16:35:19.370551 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.370531    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 08 16:35:19.370690 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.370623    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 08 16:35:19.370690 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.370661    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 08 16:35:19.379064 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.379044    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 08 16:35:19.381123 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.381104    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j9wp8"]
Jul 08 16:35:19.381240 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.381225    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:19.383535 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.383518    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 08 16:35:19.383636 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.383540    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 08 16:35:19.383709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.383693    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-26lgn\""
Jul 08 16:35:19.401306 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.401284    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65d7964d55-kbtmv"]
Jul 08 16:35:19.401437 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.401324    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rnngg"]
Jul 08 16:35:19.401437 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.401350    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j9wp8"]
Jul 08 16:35:19.401541 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.401450    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j9wp8"
Jul 08 16:35:19.404127 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.404104    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 08 16:35:19.404226 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.404194    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rr8th\""
Jul 08 16:35:19.404363 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.404345    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 08 16:35:19.404692 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.404637    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 08 16:35:19.493360 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.493328    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n22fz\" (UniqueName: \"kubernetes.io/projected/029df44a-7597-4aaa-820e-16e4e645f062-kube-api-access-n22fz\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:19.493360 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.493359    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59544202-e594-42b7-b1cc-01803b180e5d-installation-pull-secrets\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.493558 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.493384    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59544202-e594-42b7-b1cc-01803b180e5d-registry-certificates\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.493558 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.493409    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-bound-sa-token\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.493558 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.493490    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59544202-e594-42b7-b1cc-01803b180e5d-image-registry-private-configuration\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.493558 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.493532    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59544202-e594-42b7-b1cc-01803b180e5d-ca-trust-extracted\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.493739 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.493588    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029df44a-7597-4aaa-820e-16e4e645f062-tmp-dir\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:19.493739 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.493613    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llvxj\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-kube-api-access-llvxj\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.493739 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.493651    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.493739 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.493674    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/029df44a-7597-4aaa-820e-16e4e645f062-config-volume\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:19.493739 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.493710    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:19.493739 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.493730    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59544202-e594-42b7-b1cc-01803b180e5d-trusted-ca\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.595159 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595078    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-bound-sa-token\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.595159 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595126    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59544202-e594-42b7-b1cc-01803b180e5d-image-registry-private-configuration\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.595159 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595145    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59544202-e594-42b7-b1cc-01803b180e5d-ca-trust-extracted\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.595400 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595175    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029df44a-7597-4aaa-820e-16e4e645f062-tmp-dir\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:19.595400 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595206    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvxj\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-kube-api-access-llvxj\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.595400 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595249    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.595400 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595271    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/029df44a-7597-4aaa-820e-16e4e645f062-config-volume\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:19.595400 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595295    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277ceaee-f329-4dde-b0a0-d46493d0acaa-cert\") pod \"ingress-canary-j9wp8\" (UID: \"277ceaee-f329-4dde-b0a0-d46493d0acaa\") " pod="openshift-ingress-canary/ingress-canary-j9wp8"
Jul 08 16:35:19.595400 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595322    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhv9w\" (UniqueName: \"kubernetes.io/projected/277ceaee-f329-4dde-b0a0-d46493d0acaa-kube-api-access-nhv9w\") pod \"ingress-canary-j9wp8\" (UID: \"277ceaee-f329-4dde-b0a0-d46493d0acaa\") " pod="openshift-ingress-canary/ingress-canary-j9wp8"
Jul 08 16:35:19.595400 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595363    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:19.595400 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595390    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59544202-e594-42b7-b1cc-01803b180e5d-trusted-ca\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.595738 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595422    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n22fz\" (UniqueName: \"kubernetes.io/projected/029df44a-7597-4aaa-820e-16e4e645f062-kube-api-access-n22fz\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:19.595738 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595450    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59544202-e594-42b7-b1cc-01803b180e5d-installation-pull-secrets\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.595738 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595477    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59544202-e594-42b7-b1cc-01803b180e5d-registry-certificates\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.595738 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:19.595478    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:19.595738 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:19.595520    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d7964d55-kbtmv: secret "image-registry-tls" not found
Jul 08 16:35:19.595738 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:19.595594    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls podName:59544202-e594-42b7-b1cc-01803b180e5d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:20.095555386 +0000 UTC m=+35.632107865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls") pod "image-registry-65d7964d55-kbtmv" (UID: "59544202-e594-42b7-b1cc-01803b180e5d") : secret "image-registry-tls" not found
Jul 08 16:35:19.595738 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595590    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59544202-e594-42b7-b1cc-01803b180e5d-ca-trust-extracted\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.595738 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595603    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029df44a-7597-4aaa-820e-16e4e645f062-tmp-dir\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:19.596074 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:19.595895    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:19.596074 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:19.595940    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls podName:029df44a-7597-4aaa-820e-16e4e645f062 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:20.095925117 +0000 UTC m=+35.632477594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls") pod "dns-default-rnngg" (UID: "029df44a-7597-4aaa-820e-16e4e645f062") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:19.596074 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.595982    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/029df44a-7597-4aaa-820e-16e4e645f062-config-volume\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:19.596192 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.596158    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59544202-e594-42b7-b1cc-01803b180e5d-registry-certificates\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.596510 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.596479    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59544202-e594-42b7-b1cc-01803b180e5d-trusted-ca\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.600007 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.599913    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59544202-e594-42b7-b1cc-01803b180e5d-image-registry-private-configuration\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.600007 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.599971    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59544202-e594-42b7-b1cc-01803b180e5d-installation-pull-secrets\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.604176 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.604115    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llvxj\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-kube-api-access-llvxj\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.604176 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.604114    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-bound-sa-token\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:19.604325 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.604289    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n22fz\" (UniqueName: \"kubernetes.io/projected/029df44a-7597-4aaa-820e-16e4e645f062-kube-api-access-n22fz\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:19.696141 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.696107    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277ceaee-f329-4dde-b0a0-d46493d0acaa-cert\") pod \"ingress-canary-j9wp8\" (UID: \"277ceaee-f329-4dde-b0a0-d46493d0acaa\") " pod="openshift-ingress-canary/ingress-canary-j9wp8"
Jul 08 16:35:19.696141 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.696145    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhv9w\" (UniqueName: \"kubernetes.io/projected/277ceaee-f329-4dde-b0a0-d46493d0acaa-kube-api-access-nhv9w\") pod \"ingress-canary-j9wp8\" (UID: \"277ceaee-f329-4dde-b0a0-d46493d0acaa\") " pod="openshift-ingress-canary/ingress-canary-j9wp8"
Jul 08 16:35:19.698970 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.698938    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277ceaee-f329-4dde-b0a0-d46493d0acaa-cert\") pod \"ingress-canary-j9wp8\" (UID: \"277ceaee-f329-4dde-b0a0-d46493d0acaa\") " pod="openshift-ingress-canary/ingress-canary-j9wp8"
Jul 08 16:35:19.706247 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.706225    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhv9w\" (UniqueName: \"kubernetes.io/projected/277ceaee-f329-4dde-b0a0-d46493d0acaa-kube-api-access-nhv9w\") pod \"ingress-canary-j9wp8\" (UID: \"277ceaee-f329-4dde-b0a0-d46493d0acaa\") " pod="openshift-ingress-canary/ingress-canary-j9wp8"
Jul 08 16:35:19.711951 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:19.711925    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j9wp8"
Jul 08 16:35:20.022825 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:20.022737    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:20.025399 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:20.025374    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:35:20.026026 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:20.025507    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4xq57\""
Jul 08 16:35:20.099186 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:20.099155    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:20.099374 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:20.099215    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:20.099374 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:20.099308    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:20.099374 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:20.099331    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d7964d55-kbtmv: secret "image-registry-tls" not found
Jul 08 16:35:20.099525 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:20.099379    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:20.099525 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:20.099391    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls podName:59544202-e594-42b7-b1cc-01803b180e5d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:21.099373131 +0000 UTC m=+36.635925594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls") pod "image-registry-65d7964d55-kbtmv" (UID: "59544202-e594-42b7-b1cc-01803b180e5d") : secret "image-registry-tls" not found
Jul 08 16:35:20.099525 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:20.099446    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls podName:029df44a-7597-4aaa-820e-16e4e645f062 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:21.099428439 +0000 UTC m=+36.635980916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls") pod "dns-default-rnngg" (UID: "029df44a-7597-4aaa-820e-16e4e645f062") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:20.440774 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:20.440619    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j9wp8"]
Jul 08 16:35:20.487073 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:35:20.487037    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277ceaee_f329_4dde_b0a0_d46493d0acaa.slice/crio-9c1864d9d825fb94e95fefff52c833d5a04470ddcfa43d427c404a32187e2263 WatchSource:0}: Error finding container 9c1864d9d825fb94e95fefff52c833d5a04470ddcfa43d427c404a32187e2263: Status 404 returned error can't find the container with id 9c1864d9d825fb94e95fefff52c833d5a04470ddcfa43d427c404a32187e2263
Jul 08 16:35:21.023307 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:21.023181    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:21.023420 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:21.023406    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:21.025896 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:21.025866    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:35:21.026822 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:21.026588    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:35:21.026822 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:21.026588    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:35:21.026822 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:21.026610    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bxclf\""
Jul 08 16:35:21.107608 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:21.107577    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:21.107725 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:21.107640    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:21.107725 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:21.107668    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:21.107725 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:21.107683    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d7964d55-kbtmv: secret "image-registry-tls" not found
Jul 08 16:35:21.107725 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:21.107720    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:21.107926 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:21.107739    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls podName:59544202-e594-42b7-b1cc-01803b180e5d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:23.107722671 +0000 UTC m=+38.644275151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls") pod "image-registry-65d7964d55-kbtmv" (UID: "59544202-e594-42b7-b1cc-01803b180e5d") : secret "image-registry-tls" not found
Jul 08 16:35:21.107926 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:21.107760    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls podName:029df44a-7597-4aaa-820e-16e4e645f062 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:23.107750334 +0000 UTC m=+38.644302797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls") pod "dns-default-rnngg" (UID: "029df44a-7597-4aaa-820e-16e4e645f062") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:21.258754 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:21.258714    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j9wp8" event={"ID":"277ceaee-f329-4dde-b0a0-d46493d0acaa","Type":"ContainerStarted","Data":"9c1864d9d825fb94e95fefff52c833d5a04470ddcfa43d427c404a32187e2263"}
Jul 08 16:35:21.261759 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:21.261732    2550 generic.go:356] "Generic (PLEG): container finished" podID="d2c866b7-440e-4458-bfdf-fc786d235674" containerID="0902b0a71258a1b8eaa0c7df4c529434179abd4aa1adbeacfdd6645387a48666" exitCode=0
Jul 08 16:35:21.261873 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:21.261772    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp7pp" event={"ID":"d2c866b7-440e-4458-bfdf-fc786d235674","Type":"ContainerDied","Data":"0902b0a71258a1b8eaa0c7df4c529434179abd4aa1adbeacfdd6645387a48666"}
Jul 08 16:35:22.267965 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:22.267923    2550 generic.go:356] "Generic (PLEG): container finished" podID="d2c866b7-440e-4458-bfdf-fc786d235674" containerID="6dd7c1e4119a58c026ca61185f9f2610030716a5515e9ad218bf93d67a4e5606" exitCode=0
Jul 08 16:35:22.268418 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:22.267974    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp7pp" event={"ID":"d2c866b7-440e-4458-bfdf-fc786d235674","Type":"ContainerDied","Data":"6dd7c1e4119a58c026ca61185f9f2610030716a5515e9ad218bf93d67a4e5606"}
Jul 08 16:35:23.122733 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.122653    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:23.122733 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.122697    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:23.122942 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:23.122788    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:23.122942 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:23.122814    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:23.122942 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:23.122831    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d7964d55-kbtmv: secret "image-registry-tls" not found
Jul 08 16:35:23.122942 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:23.122836    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls podName:029df44a-7597-4aaa-820e-16e4e645f062 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:27.122824622 +0000 UTC m=+42.659377085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls") pod "dns-default-rnngg" (UID: "029df44a-7597-4aaa-820e-16e4e645f062") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:23.122942 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:23.122884    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls podName:59544202-e594-42b7-b1cc-01803b180e5d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:27.122862108 +0000 UTC m=+42.659414586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls") pod "image-registry-65d7964d55-kbtmv" (UID: "59544202-e594-42b7-b1cc-01803b180e5d") : secret "image-registry-tls" not found
Jul 08 16:35:23.273396 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.273248    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp7pp" event={"ID":"d2c866b7-440e-4458-bfdf-fc786d235674","Type":"ContainerStarted","Data":"7b571ad2f4c886d5c5f93214de20e3afdff9e46ad5f78c4edc8fdb2984fe266d"}
Jul 08 16:35:23.274558 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.274529    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j9wp8" event={"ID":"277ceaee-f329-4dde-b0a0-d46493d0acaa","Type":"ContainerStarted","Data":"034f976066a4fd2ae8f19208d66285e76600ca339904694f927f119299f819fc"}
Jul 08 16:35:23.299397 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.299351    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qp7pp" podStartSLOduration=8.604322463 podStartE2EDuration="38.299337228s" totalImagesPullingTime="29.695014765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:23.29796806 +0000 UTC m=+38.834520555" watchObservedRunningTime="2026-07-08 16:35:23.299337228 +0000 UTC m=+38.835889724"
Jul 08 16:35:23.316126 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.316088    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j9wp8" podStartSLOduration=2.100990489 podStartE2EDuration="4.316075217s" totalImagesPullingTime="2.215084728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:23.315507042 +0000 UTC m=+38.852059526" watchObservedRunningTime="2026-07-08 16:35:23.316075217 +0000 UTC m=+38.852627751"
Jul 08 16:35:23.526539 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.526471    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gk7dn"]
Jul 08 16:35:23.528376 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.528360    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.531074 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.531054    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 08 16:35:23.533536 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.533515    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 08 16:35:23.533679 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.533655    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 08 16:35:23.533821 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.533791    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dl4mw\""
Jul 08 16:35:23.534835 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.534813    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 08 16:35:23.551162 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.551140    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gk7dn"]
Jul 08 16:35:23.727309 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.727283    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/686acd8e-2b2b-441f-8502-cce0d131d3f3-crio-socket\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.727473 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.727317    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.727473 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.727347    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.727473 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.727440    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/686acd8e-2b2b-441f-8502-cce0d131d3f3-data-volume\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.727473 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.727466    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t24kv\" (UniqueName: \"kubernetes.io/projected/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-api-access-t24kv\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.828101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.828033    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/686acd8e-2b2b-441f-8502-cce0d131d3f3-crio-socket\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.828101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.828068    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.828101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.828085    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.828330 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.828133    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/686acd8e-2b2b-441f-8502-cce0d131d3f3-data-volume\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.828330 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.828151    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t24kv\" (UniqueName: \"kubernetes.io/projected/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-api-access-t24kv\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.828330 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:23.828156    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:23.828330 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:23.828225    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm podName:686acd8e-2b2b-441f-8502-cce0d131d3f3 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:24.328208551 +0000 UTC m=+39.864761033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gk7dn" (UID: "686acd8e-2b2b-441f-8502-cce0d131d3f3") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:23.828330 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.828226    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/686acd8e-2b2b-441f-8502-cce0d131d3f3-crio-socket\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.828330 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:23.828264    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:23.828330 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:23.828329    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls podName:686acd8e-2b2b-441f-8502-cce0d131d3f3 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:24.328313284 +0000 UTC m=+39.864865752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk7dn" (UID: "686acd8e-2b2b-441f-8502-cce0d131d3f3") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:23.828665 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.828503    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/686acd8e-2b2b-441f-8502-cce0d131d3f3-data-volume\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:23.840209 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:23.840187    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t24kv\" (UniqueName: \"kubernetes.io/projected/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-api-access-t24kv\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:24.333323 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:24.333300    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:24.333756 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:24.333328    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:24.333756 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:24.333407    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:24.333756 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:24.333446    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:24.333756 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:24.333475    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm podName:686acd8e-2b2b-441f-8502-cce0d131d3f3 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:25.333454192 +0000 UTC m=+40.870006857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gk7dn" (UID: "686acd8e-2b2b-441f-8502-cce0d131d3f3") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:24.333756 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:24.333512    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls podName:686acd8e-2b2b-441f-8502-cce0d131d3f3 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:25.333490539 +0000 UTC m=+40.870043019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk7dn" (UID: "686acd8e-2b2b-441f-8502-cce0d131d3f3") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:25.294132 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:25.294108    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4dpm5_248e79a6-fa20-4328-a4e7-0c6c54c93cd9/dns-node-resolver/0.log"
Jul 08 16:35:25.339427 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:25.339404    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:25.339748 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:25.339431    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:25.339748 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:25.339504    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:25.339748 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:25.339522    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:25.339748 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:25.339552    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm podName:686acd8e-2b2b-441f-8502-cce0d131d3f3 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:27.339540494 +0000 UTC m=+42.876092992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gk7dn" (UID: "686acd8e-2b2b-441f-8502-cce0d131d3f3") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:25.339748 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:25.339582    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls podName:686acd8e-2b2b-441f-8502-cce0d131d3f3 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:27.339572581 +0000 UTC m=+42.876125058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk7dn" (UID: "686acd8e-2b2b-441f-8502-cce0d131d3f3") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:26.569823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.569790    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-xksgk"]
Jul 08 16:35:26.602506 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.602476    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-xksgk"]
Jul 08 16:35:26.602658 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.602597    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-xksgk"
Jul 08 16:35:26.605655 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.605634    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 08 16:35:26.605655 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.605645    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 08 16:35:26.605893 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.605879    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vwvzk\""
Jul 08 16:35:26.606538 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.606525    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 08 16:35:26.606773 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.606757    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 08 16:35:26.646448 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.646425    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f2b54989-a596-495f-bad3-0b6e700ba50d-signing-cabundle\") pod \"service-ca-758f599c5-xksgk\" (UID: \"f2b54989-a596-495f-bad3-0b6e700ba50d\") " pod="openshift-service-ca/service-ca-758f599c5-xksgk"
Jul 08 16:35:26.646539 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.646455    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4vcq\" (UniqueName: \"kubernetes.io/projected/f2b54989-a596-495f-bad3-0b6e700ba50d-kube-api-access-k4vcq\") pod \"service-ca-758f599c5-xksgk\" (UID: \"f2b54989-a596-495f-bad3-0b6e700ba50d\") " pod="openshift-service-ca/service-ca-758f599c5-xksgk"
Jul 08 16:35:26.646598 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.646584    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f2b54989-a596-495f-bad3-0b6e700ba50d-signing-key\") pod \"service-ca-758f599c5-xksgk\" (UID: \"f2b54989-a596-495f-bad3-0b6e700ba50d\") " pod="openshift-service-ca/service-ca-758f599c5-xksgk"
Jul 08 16:35:26.692232 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.692215    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kxzc4_1dcc1bed-da11-489e-801a-904a8a21583a/node-ca/0.log"
Jul 08 16:35:26.746926 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.746901    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f2b54989-a596-495f-bad3-0b6e700ba50d-signing-key\") pod \"service-ca-758f599c5-xksgk\" (UID: \"f2b54989-a596-495f-bad3-0b6e700ba50d\") " pod="openshift-service-ca/service-ca-758f599c5-xksgk"
Jul 08 16:35:26.746979 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.746969    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f2b54989-a596-495f-bad3-0b6e700ba50d-signing-cabundle\") pod \"service-ca-758f599c5-xksgk\" (UID: \"f2b54989-a596-495f-bad3-0b6e700ba50d\") " pod="openshift-service-ca/service-ca-758f599c5-xksgk"
Jul 08 16:35:26.747135 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.747120    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4vcq\" (UniqueName: \"kubernetes.io/projected/f2b54989-a596-495f-bad3-0b6e700ba50d-kube-api-access-k4vcq\") pod \"service-ca-758f599c5-xksgk\" (UID: \"f2b54989-a596-495f-bad3-0b6e700ba50d\") " pod="openshift-service-ca/service-ca-758f599c5-xksgk"
Jul 08 16:35:26.747698 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.747681    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f2b54989-a596-495f-bad3-0b6e700ba50d-signing-cabundle\") pod \"service-ca-758f599c5-xksgk\" (UID: \"f2b54989-a596-495f-bad3-0b6e700ba50d\") " pod="openshift-service-ca/service-ca-758f599c5-xksgk"
Jul 08 16:35:26.749291 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.749276    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f2b54989-a596-495f-bad3-0b6e700ba50d-signing-key\") pod \"service-ca-758f599c5-xksgk\" (UID: \"f2b54989-a596-495f-bad3-0b6e700ba50d\") " pod="openshift-service-ca/service-ca-758f599c5-xksgk"
Jul 08 16:35:26.756586 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.756549    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4vcq\" (UniqueName: \"kubernetes.io/projected/f2b54989-a596-495f-bad3-0b6e700ba50d-kube-api-access-k4vcq\") pod \"service-ca-758f599c5-xksgk\" (UID: \"f2b54989-a596-495f-bad3-0b6e700ba50d\") " pod="openshift-service-ca/service-ca-758f599c5-xksgk"
Jul 08 16:35:26.912158 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:26.912117    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-xksgk"
Jul 08 16:35:27.047372 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:27.047346    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-xksgk"]
Jul 08 16:35:27.050108 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:35:27.050082    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b54989_a596_495f_bad3_0b6e700ba50d.slice/crio-1c94725d9c66d8e0c36df2dd16a3aebb0152c4d1f6dc073a06b6e3c755464899 WatchSource:0}: Error finding container 1c94725d9c66d8e0c36df2dd16a3aebb0152c4d1f6dc073a06b6e3c755464899: Status 404 returned error can't find the container with id 1c94725d9c66d8e0c36df2dd16a3aebb0152c4d1f6dc073a06b6e3c755464899
Jul 08 16:35:27.093444 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:27.093422    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j9wp8_277ceaee-f329-4dde-b0a0-d46493d0acaa/serve-healthcheck-canary/0.log"
Jul 08 16:35:27.149755 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:27.149731    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:27.149821 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:27.149783    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:27.149870 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:27.149858    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:27.149900 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:27.149873    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d7964d55-kbtmv: secret "image-registry-tls" not found
Jul 08 16:35:27.149900 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:27.149880    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:27.149953 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:27.149913    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls podName:59544202-e594-42b7-b1cc-01803b180e5d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:35.149901086 +0000 UTC m=+50.686453549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls") pod "image-registry-65d7964d55-kbtmv" (UID: "59544202-e594-42b7-b1cc-01803b180e5d") : secret "image-registry-tls" not found
Jul 08 16:35:27.149953 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:27.149926    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls podName:029df44a-7597-4aaa-820e-16e4e645f062 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:35.14991931 +0000 UTC m=+50.686471774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls") pod "dns-default-rnngg" (UID: "029df44a-7597-4aaa-820e-16e4e645f062") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:27.284083 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:27.284062    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-xksgk" event={"ID":"f2b54989-a596-495f-bad3-0b6e700ba50d","Type":"ContainerStarted","Data":"1c94725d9c66d8e0c36df2dd16a3aebb0152c4d1f6dc073a06b6e3c755464899"}
Jul 08 16:35:27.351678 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:27.351659    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:27.351740 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:27.351685    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:27.351781 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:27.351772    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:27.351848 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:27.351835    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:27.351881 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:27.351822    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm podName:686acd8e-2b2b-441f-8502-cce0d131d3f3 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:31.351807002 +0000 UTC m=+46.888359479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gk7dn" (UID: "686acd8e-2b2b-441f-8502-cce0d131d3f3") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:27.351881 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:27.351876    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls podName:686acd8e-2b2b-441f-8502-cce0d131d3f3 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:31.35186609 +0000 UTC m=+46.888418568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk7dn" (UID: "686acd8e-2b2b-441f-8502-cce0d131d3f3") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:30.296358 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:30.296323    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-xksgk" event={"ID":"f2b54989-a596-495f-bad3-0b6e700ba50d","Type":"ContainerStarted","Data":"e2e671e26d186f10924fb6703e0aca2bce0f36b7ee8dc11341123060cb8a4cd3"}
Jul 08 16:35:30.314032 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:30.313982    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-xksgk" podStartSLOduration=2.158243625 podStartE2EDuration="4.313964544s" totalImagesPullingTime="2.155720919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:30.313314284 +0000 UTC m=+45.849866767" watchObservedRunningTime="2026-07-08 16:35:30.313964544 +0000 UTC m=+45.850517029"
Jul 08 16:35:31.382849 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:31.382824    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:31.382849 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:31.382852    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:31.383308 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:31.382981    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:31.383308 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:35:31.383030    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls podName:686acd8e-2b2b-441f-8502-cce0d131d3f3 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:39.383016269 +0000 UTC m=+54.919568745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk7dn" (UID: "686acd8e-2b2b-441f-8502-cce0d131d3f3") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:31.383375 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:31.383345    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/686acd8e-2b2b-441f-8502-cce0d131d3f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:35.210511 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:35.210474    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:35.211000 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:35.210587    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:35.212846 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:35.212824    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029df44a-7597-4aaa-820e-16e4e645f062-metrics-tls\") pod \"dns-default-rnngg\" (UID: \"029df44a-7597-4aaa-820e-16e4e645f062\") " pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:35.212965 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:35.212881    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls\") pod \"image-registry-65d7964d55-kbtmv\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") " pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:35.279616 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:35.279587    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:35.292793 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:35.292767    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:35.429875 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:35.429827    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65d7964d55-kbtmv"]
Jul 08 16:35:35.432335 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:35:35.432308    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59544202_e594_42b7_b1cc_01803b180e5d.slice/crio-416e33d3e4781a4be6224230763e54022c860a224359a4a18c85561b51fb7123 WatchSource:0}: Error finding container 416e33d3e4781a4be6224230763e54022c860a224359a4a18c85561b51fb7123: Status 404 returned error can't find the container with id 416e33d3e4781a4be6224230763e54022c860a224359a4a18c85561b51fb7123
Jul 08 16:35:35.442477 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:35.442454    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rnngg"]
Jul 08 16:35:35.444541 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:35:35.444523    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029df44a_7597_4aaa_820e_16e4e645f062.slice/crio-ca1c2833634b66db04eceb23f4ad019b222fbd4d621c5fd64940cdc90c92f19e WatchSource:0}: Error finding container ca1c2833634b66db04eceb23f4ad019b222fbd4d621c5fd64940cdc90c92f19e: Status 404 returned error can't find the container with id ca1c2833634b66db04eceb23f4ad019b222fbd4d621c5fd64940cdc90c92f19e
Jul 08 16:35:36.311978 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:36.311944    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d7964d55-kbtmv" event={"ID":"59544202-e594-42b7-b1cc-01803b180e5d","Type":"ContainerStarted","Data":"1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e"}
Jul 08 16:35:36.312409 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:36.312004    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d7964d55-kbtmv" event={"ID":"59544202-e594-42b7-b1cc-01803b180e5d","Type":"ContainerStarted","Data":"416e33d3e4781a4be6224230763e54022c860a224359a4a18c85561b51fb7123"}
Jul 08 16:35:36.312409 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:36.312178    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:36.313493 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:36.313436    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rnngg" event={"ID":"029df44a-7597-4aaa-820e-16e4e645f062","Type":"ContainerStarted","Data":"ca1c2833634b66db04eceb23f4ad019b222fbd4d621c5fd64940cdc90c92f19e"}
Jul 08 16:35:36.332826 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:36.332777    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65d7964d55-kbtmv" podStartSLOduration=54.332761184 podStartE2EDuration="54.332761184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:36.331887388 +0000 UTC m=+51.868439874" watchObservedRunningTime="2026-07-08 16:35:36.332761184 +0000 UTC m=+51.869313671"
Jul 08 16:35:38.323576 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:38.323530    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rnngg" event={"ID":"029df44a-7597-4aaa-820e-16e4e645f062","Type":"ContainerStarted","Data":"ce578e6357b86231279eb1de658df83a6edbde14ba816a7f3d43e6f965da508f"}
Jul 08 16:35:38.323960 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:38.323631    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rnngg" event={"ID":"029df44a-7597-4aaa-820e-16e4e645f062","Type":"ContainerStarted","Data":"8252d5a0d676f06b5891f80407f584ec06bb8587f73bfbc862b7720f660ac2cc"}
Jul 08 16:35:38.323960 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:38.323704    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:38.340743 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:38.340706    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rnngg" podStartSLOduration=17.073629543 podStartE2EDuration="19.340694903s" totalImagesPullingTime="2.26706536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:38.340352811 +0000 UTC m=+53.876905297" watchObservedRunningTime="2026-07-08 16:35:38.340694903 +0000 UTC m=+53.877247387"
Jul 08 16:35:39.443363 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:39.443339    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:39.445503 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:39.445487    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/686acd8e-2b2b-441f-8502-cce0d131d3f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk7dn\" (UID: \"686acd8e-2b2b-441f-8502-cce0d131d3f3\") " pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:39.737965 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:39.737908    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gk7dn"
Jul 08 16:35:39.876646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:39.876619    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gk7dn"]
Jul 08 16:35:39.879823 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:35:39.879794    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686acd8e_2b2b_441f_8502_cce0d131d3f3.slice/crio-b593b16d0f2e5e4afc8dcb12516d47b93e2f330d1c21a84fe496ca5ece08833a WatchSource:0}: Error finding container b593b16d0f2e5e4afc8dcb12516d47b93e2f330d1c21a84fe496ca5ece08833a: Status 404 returned error can't find the container with id b593b16d0f2e5e4afc8dcb12516d47b93e2f330d1c21a84fe496ca5ece08833a
Jul 08 16:35:40.328655 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:40.328622    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk7dn" event={"ID":"686acd8e-2b2b-441f-8502-cce0d131d3f3","Type":"ContainerStarted","Data":"b593b16d0f2e5e4afc8dcb12516d47b93e2f330d1c21a84fe496ca5ece08833a"}
Jul 08 16:35:44.341947 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:44.341916    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk7dn" event={"ID":"686acd8e-2b2b-441f-8502-cce0d131d3f3","Type":"ContainerStarted","Data":"7b7e950614a012d6b4207783e9549021dbd6d3dc7a8ce5dd4b489d5bac053ac6"}
Jul 08 16:35:45.254221 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:45.254192    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzzb2"
Jul 08 16:35:45.347353 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:45.347321    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk7dn" event={"ID":"686acd8e-2b2b-441f-8502-cce0d131d3f3","Type":"ContainerStarted","Data":"47042dcdf99ca021d46e8290fa6af808b1bd625d208e366bafd11f34465b2e24"}
Jul 08 16:35:47.355941 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:47.355903    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk7dn" event={"ID":"686acd8e-2b2b-441f-8502-cce0d131d3f3","Type":"ContainerStarted","Data":"e7ee48cf6426ebaaa918a91b7017c0e2f8bc27a2ed6de34c7aa91d15b872012a"}
Jul 08 16:35:47.374462 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:47.374423    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gk7dn" podStartSLOduration=17.560016782 podStartE2EDuration="24.374408936s" totalImagesPullingTime="6.814392154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:35:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:47.374341113 +0000 UTC m=+62.910893598" watchObservedRunningTime="2026-07-08 16:35:47.374408936 +0000 UTC m=+62.910961421"
Jul 08 16:35:47.902926 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:47.902889    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65d7964d55-kbtmv"]
Jul 08 16:35:48.327452 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:48.327423    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rnngg"
Jul 08 16:35:50.724451 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:50.724419    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:50.724858 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:50.724455    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgnv\" (UniqueName: \"kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv\") pod \"network-check-target-79p9f\" (UID: \"d22dbf53-7ebd-4312-a491-894b8392147d\") " pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:50.727655 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:50.727631    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:35:50.730574 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:50.730549    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:35:50.737844 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:50.737829    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:35:50.741089 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:50.741068    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/998f1936-ebcf-489b-86d0-e6ff22c6f840-original-pull-secret\") pod \"global-pull-secret-syncer-gm9kt\" (UID: \"998f1936-ebcf-489b-86d0-e6ff22c6f840\") " pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:50.748066 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:50.748042    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgnv\" (UniqueName: \"kubernetes.io/projected/d22dbf53-7ebd-4312-a491-894b8392147d-kube-api-access-4sgnv\") pod \"network-check-target-79p9f\" (UID: \"d22dbf53-7ebd-4312-a491-894b8392147d\") " pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:50.825440 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:50.825410    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:50.828197 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:50.828182    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:35:50.837948 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:50.837926    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787b3d58-fe95-4bac-a035-bd65399a7f30-metrics-certs\") pod \"network-metrics-daemon-xphrq\" (UID: \"787b3d58-fe95-4bac-a035-bd65399a7f30\") " pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:50.935681 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:50.935662    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4xq57\""
Jul 08 16:35:50.943554 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:50.943540    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xphrq"
Jul 08 16:35:51.035114 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.035087    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm9kt"
Jul 08 16:35:51.044275 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.044256    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bxclf\""
Jul 08 16:35:51.052239 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.052220    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:51.059470 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.059446    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xphrq"]
Jul 08 16:35:51.061947 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:35:51.061920    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787b3d58_fe95_4bac_a035_bd65399a7f30.slice/crio-1d42093be7842dbe76241b00e74d69b1e86f92479738906c52a12a85bb50e601 WatchSource:0}: Error finding container 1d42093be7842dbe76241b00e74d69b1e86f92479738906c52a12a85bb50e601: Status 404 returned error can't find the container with id 1d42093be7842dbe76241b00e74d69b1e86f92479738906c52a12a85bb50e601
Jul 08 16:35:51.161119 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.161089    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gm9kt"]
Jul 08 16:35:51.163692 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:35:51.163663    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998f1936_ebcf_489b_86d0_e6ff22c6f840.slice/crio-3123b0285bb45ed9602b647b726ce0a5f861a6d0063d0a7afceeb2d9a2d779df WatchSource:0}: Error finding container 3123b0285bb45ed9602b647b726ce0a5f861a6d0063d0a7afceeb2d9a2d779df: Status 404 returned error can't find the container with id 3123b0285bb45ed9602b647b726ce0a5f861a6d0063d0a7afceeb2d9a2d779df
Jul 08 16:35:51.180320 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.180298    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-79p9f"]
Jul 08 16:35:51.183404 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:35:51.183382    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22dbf53_7ebd_4312_a491_894b8392147d.slice/crio-a04d39eed1c084d99c7e548aadc1a282cf8e744d2f14e1e1e457ef700f12f3b6 WatchSource:0}: Error finding container a04d39eed1c084d99c7e548aadc1a282cf8e744d2f14e1e1e457ef700f12f3b6: Status 404 returned error can't find the container with id a04d39eed1c084d99c7e548aadc1a282cf8e744d2f14e1e1e457ef700f12f3b6
Jul 08 16:35:51.370301 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.370273    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xphrq" event={"ID":"787b3d58-fe95-4bac-a035-bd65399a7f30","Type":"ContainerStarted","Data":"1d42093be7842dbe76241b00e74d69b1e86f92479738906c52a12a85bb50e601"}
Jul 08 16:35:51.371377 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.371356    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gm9kt" event={"ID":"998f1936-ebcf-489b-86d0-e6ff22c6f840","Type":"ContainerStarted","Data":"3123b0285bb45ed9602b647b726ce0a5f861a6d0063d0a7afceeb2d9a2d779df"}
Jul 08 16:35:51.372432 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.372413    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-79p9f" event={"ID":"d22dbf53-7ebd-4312-a491-894b8392147d","Type":"ContainerStarted","Data":"a04d39eed1c084d99c7e548aadc1a282cf8e744d2f14e1e1e457ef700f12f3b6"}
Jul 08 16:35:51.990964 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.990061    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8468f948d5-x6qg9"]
Jul 08 16:35:51.993643 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.992837    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:51.995421 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.995392    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fpsds\""
Jul 08 16:35:51.995699 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.995680    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 16:35:51.996312 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.996294    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 16:35:51.996544 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.996519    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 16:35:51.996729 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.996714    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 16:35:51.997196 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.996901    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 16:35:51.997196 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.996986    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 16:35:51.997196 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:51.996902    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 16:35:52.004377 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.004354    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8468f948d5-x6qg9"]
Jul 08 16:35:52.033121 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.033093    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnnm\" (UniqueName: \"kubernetes.io/projected/6b23c066-a365-4e6e-8170-93e0e2bc999d-kube-api-access-jrnnm\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.033234 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.033141    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-oauth-serving-cert\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.033234 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.033177    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-serving-cert\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.033234 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.033220    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-config\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.033395 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.033258    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-service-ca\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.033395 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.033274    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-oauth-config\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.134445 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.134409    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnnm\" (UniqueName: \"kubernetes.io/projected/6b23c066-a365-4e6e-8170-93e0e2bc999d-kube-api-access-jrnnm\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.134646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.134463    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-oauth-serving-cert\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.134646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.134493    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-serving-cert\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.134646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.134527    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-config\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.134646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.134554    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-service-ca\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.134646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.134600    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-oauth-config\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.135557 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.135530    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-oauth-serving-cert\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.135557 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.135543    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-config\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.135734 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.135682    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-service-ca\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.138285 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.138246    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-serving-cert\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.138797 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.138763    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-oauth-config\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.147179 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.147148    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnnm\" (UniqueName: \"kubernetes.io/projected/6b23c066-a365-4e6e-8170-93e0e2bc999d-kube-api-access-jrnnm\") pod \"console-8468f948d5-x6qg9\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") " pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.308756 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.308727    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:35:52.382805 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.382723    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xphrq" event={"ID":"787b3d58-fe95-4bac-a035-bd65399a7f30","Type":"ContainerStarted","Data":"b1a879416a34ffea65ff888f4bc8fb883e25924206b778f6648d1893f7f464d6"}
Jul 08 16:35:52.470033 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:52.469997    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8468f948d5-x6qg9"]
Jul 08 16:35:52.476928 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:35:52.476888    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b23c066_a365_4e6e_8170_93e0e2bc999d.slice/crio-30c17d16c360578eb75cdfc26fe91bae6a3c98d5db94ea89778b9869cb7820a2 WatchSource:0}: Error finding container 30c17d16c360578eb75cdfc26fe91bae6a3c98d5db94ea89778b9869cb7820a2: Status 404 returned error can't find the container with id 30c17d16c360578eb75cdfc26fe91bae6a3c98d5db94ea89778b9869cb7820a2
Jul 08 16:35:53.389594 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:53.389541    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xphrq" event={"ID":"787b3d58-fe95-4bac-a035-bd65399a7f30","Type":"ContainerStarted","Data":"e745746dd6aa516b0311e95e06a75704577950ea935d4a613bb911e66f128b2f"}
Jul 08 16:35:53.390875 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:53.390844    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8468f948d5-x6qg9" event={"ID":"6b23c066-a365-4e6e-8170-93e0e2bc999d","Type":"ContainerStarted","Data":"30c17d16c360578eb75cdfc26fe91bae6a3c98d5db94ea89778b9869cb7820a2"}
Jul 08 16:35:53.406356 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:53.406299    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xphrq" podStartSLOduration=67.259456019 podStartE2EDuration="1m8.406285521s" totalImagesPullingTime="1.146829502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:53.405666558 +0000 UTC m=+68.942219042" watchObservedRunningTime="2026-07-08 16:35:53.406285521 +0000 UTC m=+68.942838005"
Jul 08 16:35:57.908031 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:57.908003    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:35:58.410143 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.410107    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-79p9f" event={"ID":"d22dbf53-7ebd-4312-a491-894b8392147d","Type":"ContainerStarted","Data":"73a6033d673a7707a7abac2a782a3c7690677dbbf03ef60cf139f3cc1e9bdabc"}
Jul 08 16:35:58.410308 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.410272    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:35:58.411661 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.411638    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8468f948d5-x6qg9" event={"ID":"6b23c066-a365-4e6e-8170-93e0e2bc999d","Type":"ContainerStarted","Data":"d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118"}
Jul 08 16:35:58.412867 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.412846    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gm9kt" event={"ID":"998f1936-ebcf-489b-86d0-e6ff22c6f840","Type":"ContainerStarted","Data":"4f83fe3e49f61903a05ffdfcec190d60062fb95064d25f01331b4fd4addee6c8"}
Jul 08 16:35:58.424870 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.424822    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-79p9f" podStartSLOduration=67.232006371 podStartE2EDuration="1m13.424809456s" totalImagesPullingTime="6.192803085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:58.42394432 +0000 UTC m=+73.960496802" watchObservedRunningTime="2026-07-08 16:35:58.424809456 +0000 UTC m=+73.961361939"
Jul 08 16:35:58.439416 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.439381    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gm9kt" podStartSLOduration=67.215396118 podStartE2EDuration="1m13.439369344s" totalImagesPullingTime="6.223973226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:58.43831132 +0000 UTC m=+73.974863803" watchObservedRunningTime="2026-07-08 16:35:58.439369344 +0000 UTC m=+73.975921829"
Jul 08 16:35:58.454148 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.454107    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8468f948d5-x6qg9" podStartSLOduration=2.5494721 podStartE2EDuration="7.454095521s" totalImagesPullingTime="4.904623421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:58.453647165 +0000 UTC m=+73.990199651" watchObservedRunningTime="2026-07-08 16:35:58.454095521 +0000 UTC m=+73.990648005"
Jul 08 16:35:58.874871 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.874842    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"]
Jul 08 16:35:58.877422 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.877407    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:58.879749 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.879728    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 08 16:35:58.879749 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.879743    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 08 16:35:58.879920 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.879790    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 08 16:35:58.880593 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.880551    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 08 16:35:58.880693 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.880631    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 08 16:35:58.880693 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.880635    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zh7vm\""
Jul 08 16:35:58.887691 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.887665    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"]
Jul 08 16:35:58.983674 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.983650    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/187dd64b-3c5d-4a28-bc2f-eaf1fec6deed-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-qdvpc\" (UID: \"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:58.983953 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.983683    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s87ww\" (UniqueName: \"kubernetes.io/projected/187dd64b-3c5d-4a28-bc2f-eaf1fec6deed-kube-api-access-s87ww\") pod \"prometheus-operator-567f7b8b7c-qdvpc\" (UID: \"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:58.983953 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.983702    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/187dd64b-3c5d-4a28-bc2f-eaf1fec6deed-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qdvpc\" (UID: \"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:58.983953 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:58.983720    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/187dd64b-3c5d-4a28-bc2f-eaf1fec6deed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-qdvpc\" (UID: \"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:59.084238 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:59.084205    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/187dd64b-3c5d-4a28-bc2f-eaf1fec6deed-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qdvpc\" (UID: \"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:59.084238 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:59.084247    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/187dd64b-3c5d-4a28-bc2f-eaf1fec6deed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-qdvpc\" (UID: \"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:59.084448 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:59.084292    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/187dd64b-3c5d-4a28-bc2f-eaf1fec6deed-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-qdvpc\" (UID: \"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:59.084448 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:59.084317    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s87ww\" (UniqueName: \"kubernetes.io/projected/187dd64b-3c5d-4a28-bc2f-eaf1fec6deed-kube-api-access-s87ww\") pod \"prometheus-operator-567f7b8b7c-qdvpc\" (UID: \"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:59.084993 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:59.084970    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/187dd64b-3c5d-4a28-bc2f-eaf1fec6deed-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-qdvpc\" (UID: \"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:59.086699 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:59.086669    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/187dd64b-3c5d-4a28-bc2f-eaf1fec6deed-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qdvpc\" (UID: \"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:59.086804 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:59.086705    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/187dd64b-3c5d-4a28-bc2f-eaf1fec6deed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-qdvpc\" (UID: \"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:59.092514 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:59.092490    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s87ww\" (UniqueName: \"kubernetes.io/projected/187dd64b-3c5d-4a28-bc2f-eaf1fec6deed-kube-api-access-s87ww\") pod \"prometheus-operator-567f7b8b7c-qdvpc\" (UID: \"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:59.187656 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:59.187584    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"
Jul 08 16:35:59.314703 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:59.314673    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc"]
Jul 08 16:35:59.317673 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:35:59.317649    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187dd64b_3c5d_4a28_bc2f_eaf1fec6deed.slice/crio-3c63b3ff326459a09b930e2061d57f5025862fbcc2febc6abb89531e9d1fc8b5 WatchSource:0}: Error finding container 3c63b3ff326459a09b930e2061d57f5025862fbcc2febc6abb89531e9d1fc8b5: Status 404 returned error can't find the container with id 3c63b3ff326459a09b930e2061d57f5025862fbcc2febc6abb89531e9d1fc8b5
Jul 08 16:35:59.417448 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:35:59.417422    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc" event={"ID":"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed","Type":"ContainerStarted","Data":"3c63b3ff326459a09b930e2061d57f5025862fbcc2febc6abb89531e9d1fc8b5"}
Jul 08 16:36:01.425849 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:01.425818    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc" event={"ID":"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed","Type":"ContainerStarted","Data":"c668b6f91ba134d8ed23188939b1cc05c9efdf2d138a19bc89d9ac82ad52a3d8"}
Jul 08 16:36:02.309622 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:02.309590    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:36:02.309798 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:02.309640    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:36:02.314053 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:02.314033    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:36:02.431181 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:02.431150    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc" event={"ID":"187dd64b-3c5d-4a28-bc2f-eaf1fec6deed","Type":"ContainerStarted","Data":"c841246cad0f20e95706f213a4eca0398a366e3df95adf8290d1031809d468b8"}
Jul 08 16:36:02.434676 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:02.434658    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:36:02.448034 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:02.447985    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qdvpc" podStartSLOduration=2.419563641 podStartE2EDuration="4.44797052s" totalImagesPullingTime="2.028406879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:02.447781218 +0000 UTC m=+77.984333720" watchObservedRunningTime="2026-07-08 16:36:02.44797052 +0000 UTC m=+77.984523006"
Jul 08 16:36:04.261647 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.261593    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"]
Jul 08 16:36:04.265101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.265084    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.267709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.267684    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 08 16:36:04.267825 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.267708    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 08 16:36:04.267825 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.267722    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wcjdv\""
Jul 08 16:36:04.275604 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.275581    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"]
Jul 08 16:36:04.290640 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.290617    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"]
Jul 08 16:36:04.293521 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.293505    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.296013 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.295996    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 08 16:36:04.296103 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.296012    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sxqll\""
Jul 08 16:36:04.296751 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.296244    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 08 16:36:04.296751 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.296247    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 08 16:36:04.296751 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.296275    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bhf4t"]
Jul 08 16:36:04.299473 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.299454    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.301817 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.301797    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 08 16:36:04.301915 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.301886    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7q6bj\""
Jul 08 16:36:04.302066 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.302052    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 08 16:36:04.302372 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.302139    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 08 16:36:04.304783 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.304763    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"]
Jul 08 16:36:04.318485 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.318464    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfea924a-5429-4fbd-84f1-089e2d2a253c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.318592 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.318520    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfea924a-5429-4fbd-84f1-089e2d2a253c-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.318592 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.318551    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfea924a-5429-4fbd-84f1-089e2d2a253c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.318706 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.318625    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kff6c\" (UniqueName: \"kubernetes.io/projected/dfea924a-5429-4fbd-84f1-089e2d2a253c-kube-api-access-kff6c\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.419596 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419548    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8487b755-cece-4303-8108-15623916d5de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.419737 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419614    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8487b755-cece-4303-8108-15623916d5de-node-exporter-wtmp\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.419737 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419644    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcnwh\" (UniqueName: \"kubernetes.io/projected/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-api-access-gcnwh\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.419737 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419673    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfea924a-5429-4fbd-84f1-089e2d2a253c-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.419737 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419700    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.419737 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419730    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfea924a-5429-4fbd-84f1-089e2d2a253c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.419983 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419754    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8487b755-cece-4303-8108-15623916d5de-metrics-client-ca\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.419983 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419821    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8487b755-cece-4303-8108-15623916d5de-node-exporter-textfile\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.419983 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419864    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.419983 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:36:04.419875    2550 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 08 16:36:04.419983 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419894    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8487b755-cece-4303-8108-15623916d5de-sys\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.419983 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419926    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kff6c\" (UniqueName: \"kubernetes.io/projected/dfea924a-5429-4fbd-84f1-089e2d2a253c-kube-api-access-kff6c\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.419983 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:36:04.419943    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfea924a-5429-4fbd-84f1-089e2d2a253c-openshift-state-metrics-tls podName:dfea924a-5429-4fbd-84f1-089e2d2a253c nodeName:}" failed. No retries permitted until 2026-07-08 16:36:04.919927295 +0000 UTC m=+80.456479758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/dfea924a-5429-4fbd-84f1-089e2d2a253c-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-mmxcv" (UID: "dfea924a-5429-4fbd-84f1-089e2d2a253c") : secret "openshift-state-metrics-tls" not found
Jul 08 16:36:04.419983 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419970    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8487b755-cece-4303-8108-15623916d5de-node-exporter-tls\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.420341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.419991    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8487b755-cece-4303-8108-15623916d5de-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.420341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.420017    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24jwp\" (UniqueName: \"kubernetes.io/projected/8487b755-cece-4303-8108-15623916d5de-kube-api-access-24jwp\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.420341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.420040    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfea924a-5429-4fbd-84f1-089e2d2a253c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.420341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.420073    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.420341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.420096    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.420341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.420115    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.420341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.420134    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8487b755-cece-4303-8108-15623916d5de-root\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.420679 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.420464    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfea924a-5429-4fbd-84f1-089e2d2a253c-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.422687 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.422667    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfea924a-5429-4fbd-84f1-089e2d2a253c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.434251 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.434221    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kff6c\" (UniqueName: \"kubernetes.io/projected/dfea924a-5429-4fbd-84f1-089e2d2a253c-kube-api-access-kff6c\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.520949 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.520867    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.520949 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.520916    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.520949 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.520946    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.521186 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.520979    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8487b755-cece-4303-8108-15623916d5de-root\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.521186 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521014    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8487b755-cece-4303-8108-15623916d5de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.521186 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521046    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8487b755-cece-4303-8108-15623916d5de-node-exporter-wtmp\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.521186 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:36:04.521057    2550 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 08 16:36:04.521186 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:36:04.521121    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-state-metrics-tls podName:8edb418c-0c8e-41f4-b442-a5e6dcefb2e1 nodeName:}" failed. No retries permitted until 2026-07-08 16:36:05.021101234 +0000 UTC m=+80.557653715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-ljv42" (UID: "8edb418c-0c8e-41f4-b442-a5e6dcefb2e1") : secret "kube-state-metrics-tls" not found
Jul 08 16:36:04.521186 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521135    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8487b755-cece-4303-8108-15623916d5de-root\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.521462 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521366    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8487b755-cece-4303-8108-15623916d5de-node-exporter-wtmp\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.521462 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521429    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnwh\" (UniqueName: \"kubernetes.io/projected/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-api-access-gcnwh\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.521557 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521465    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.521557 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521537    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8487b755-cece-4303-8108-15623916d5de-metrics-client-ca\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.521688 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521607    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8487b755-cece-4303-8108-15623916d5de-node-exporter-textfile\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.521688 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521662    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.521784 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521692    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8487b755-cece-4303-8108-15623916d5de-sys\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.521784 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521743    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8487b755-cece-4303-8108-15623916d5de-node-exporter-tls\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.521784 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521770    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8487b755-cece-4303-8108-15623916d5de-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.521928 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.521800    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24jwp\" (UniqueName: \"kubernetes.io/projected/8487b755-cece-4303-8108-15623916d5de-kube-api-access-24jwp\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.522398 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.522368    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8487b755-cece-4303-8108-15623916d5de-node-exporter-textfile\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.522542 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.522516    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.522629 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.522616    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8487b755-cece-4303-8108-15623916d5de-sys\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.522855 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.522837    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.522957 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.522874    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.523044 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:36:04.522957    2550 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 08 16:36:04.523166 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:36:04.523154    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8487b755-cece-4303-8108-15623916d5de-node-exporter-tls podName:8487b755-cece-4303-8108-15623916d5de nodeName:}" failed. No retries permitted until 2026-07-08 16:36:05.023136011 +0000 UTC m=+80.559688488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8487b755-cece-4303-8108-15623916d5de-node-exporter-tls") pod "node-exporter-bhf4t" (UID: "8487b755-cece-4303-8108-15623916d5de") : secret "node-exporter-tls" not found
Jul 08 16:36:04.523256 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.523016    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8487b755-cece-4303-8108-15623916d5de-metrics-client-ca\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.523444 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.523419    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8487b755-cece-4303-8108-15623916d5de-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.525320 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.525301    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8487b755-cece-4303-8108-15623916d5de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.525409 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.525350    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.530609 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.530548    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcnwh\" (UniqueName: \"kubernetes.io/projected/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-api-access-gcnwh\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:04.531005 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.530987    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24jwp\" (UniqueName: \"kubernetes.io/projected/8487b755-cece-4303-8108-15623916d5de-kube-api-access-24jwp\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:04.925847 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.925827    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfea924a-5429-4fbd-84f1-089e2d2a253c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:04.927887 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:04.927866    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfea924a-5429-4fbd-84f1-089e2d2a253c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mmxcv\" (UID: \"dfea924a-5429-4fbd-84f1-089e2d2a253c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:05.026033 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.026014    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8487b755-cece-4303-8108-15623916d5de-node-exporter-tls\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:05.026127 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.026058    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:05.028189 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.028164    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8edb418c-0c8e-41f4-b442-a5e6dcefb2e1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-ljv42\" (UID: \"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:05.028263 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.028237    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8487b755-cece-4303-8108-15623916d5de-node-exporter-tls\") pod \"node-exporter-bhf4t\" (UID: \"8487b755-cece-4303-8108-15623916d5de\") " pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:05.174792 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.174768    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"
Jul 08 16:36:05.208461 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.208433    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"
Jul 08 16:36:05.215203 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.213965    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bhf4t"
Jul 08 16:36:05.233233 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:36:05.233197    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8487b755_cece_4303_8108_15623916d5de.slice/crio-88a4fc2ce8ec3aa8d5e10a6a4c853180064f6a330509225b9dd6b4ded2ea631a WatchSource:0}: Error finding container 88a4fc2ce8ec3aa8d5e10a6a4c853180064f6a330509225b9dd6b4ded2ea631a: Status 404 returned error can't find the container with id 88a4fc2ce8ec3aa8d5e10a6a4c853180064f6a330509225b9dd6b4ded2ea631a
Jul 08 16:36:05.351212 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.350477    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv"]
Jul 08 16:36:05.356126 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:36:05.355876    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfea924a_5429_4fbd_84f1_089e2d2a253c.slice/crio-91a8f63a5388b8c754b898bdb92f4b1d509f70aa448a0d569b11e1535964f468 WatchSource:0}: Error finding container 91a8f63a5388b8c754b898bdb92f4b1d509f70aa448a0d569b11e1535964f468: Status 404 returned error can't find the container with id 91a8f63a5388b8c754b898bdb92f4b1d509f70aa448a0d569b11e1535964f468
Jul 08 16:36:05.390134 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.390107    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42"]
Jul 08 16:36:05.392618 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:36:05.392594    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edb418c_0c8e_41f4_b442_a5e6dcefb2e1.slice/crio-0bb48c44d951cb78c149f3417df5cfced14073176a061732e31389653f498238 WatchSource:0}: Error finding container 0bb48c44d951cb78c149f3417df5cfced14073176a061732e31389653f498238: Status 404 returned error can't find the container with id 0bb48c44d951cb78c149f3417df5cfced14073176a061732e31389653f498238
Jul 08 16:36:05.450598 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.450546    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv" event={"ID":"dfea924a-5429-4fbd-84f1-089e2d2a253c","Type":"ContainerStarted","Data":"095fbcf2bd37a0c0b1a6959c90f4c6b8dbbfa7d400dfb2ad640cfcca8ff9971a"}
Jul 08 16:36:05.450698 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.450665    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv" event={"ID":"dfea924a-5429-4fbd-84f1-089e2d2a253c","Type":"ContainerStarted","Data":"91a8f63a5388b8c754b898bdb92f4b1d509f70aa448a0d569b11e1535964f468"}
Jul 08 16:36:05.452594 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.452437    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42" event={"ID":"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1","Type":"ContainerStarted","Data":"0bb48c44d951cb78c149f3417df5cfced14073176a061732e31389653f498238"}
Jul 08 16:36:05.453646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:05.453610    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhf4t" event={"ID":"8487b755-cece-4303-8108-15623916d5de","Type":"ContainerStarted","Data":"88a4fc2ce8ec3aa8d5e10a6a4c853180064f6a330509225b9dd6b4ded2ea631a"}
Jul 08 16:36:06.462822 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:06.462788    2550 generic.go:356] "Generic (PLEG): container finished" podID="8487b755-cece-4303-8108-15623916d5de" containerID="2c383178278ee2a750b5567d7a36f84d674a2f0016e37657e9fe0f34d3cba20d" exitCode=0
Jul 08 16:36:06.463187 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:06.462888    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhf4t" event={"ID":"8487b755-cece-4303-8108-15623916d5de","Type":"ContainerDied","Data":"2c383178278ee2a750b5567d7a36f84d674a2f0016e37657e9fe0f34d3cba20d"}
Jul 08 16:36:06.465656 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:06.465629    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv" event={"ID":"dfea924a-5429-4fbd-84f1-089e2d2a253c","Type":"ContainerStarted","Data":"798a5db779ca601cdae592b691f361088459bbbbd907f18075ce1a955a6bcd92"}
Jul 08 16:36:07.244909 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.244822    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-96897fc96-xrbqn"]
Jul 08 16:36:07.248598 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.248556    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.251645 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.251383    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 08 16:36:07.251645 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.251399    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 08 16:36:07.251645 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.251440    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-42zzq\""
Jul 08 16:36:07.251645 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.251383    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 08 16:36:07.251872 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.251761    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4422dsfpusbcc\""
Jul 08 16:36:07.251872 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.251853    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 08 16:36:07.252122 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.252096    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 08 16:36:07.265829 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.265747    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-96897fc96-xrbqn"]
Jul 08 16:36:07.346275 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.346247    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-tls\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.346415 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.346296    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-grpc-tls\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.346415 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.346329    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.346415 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.346365    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.346415 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.346395    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.346630 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.346419    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbrc\" (UniqueName: \"kubernetes.io/projected/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-kube-api-access-zgbrc\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.346630 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.346468    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.346630 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.346499    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-metrics-client-ca\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.447591 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.447546    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.447706 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.447596    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbrc\" (UniqueName: \"kubernetes.io/projected/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-kube-api-access-zgbrc\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.447706 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.447641    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.447706 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.447675    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-metrics-client-ca\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.447860 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.447710    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-tls\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.447860 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.447752    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-grpc-tls\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.447860 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.447788    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.447860 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.447829    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.448730 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.448677    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-metrics-client-ca\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.450980 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.450955    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.451745 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.451283    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.451745 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.451659    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-grpc-tls\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.452149 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.452122    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-tls\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.452308 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.452285    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.452479 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.452459    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.456493 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.456468    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbrc\" (UniqueName: \"kubernetes.io/projected/6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30-kube-api-access-zgbrc\") pod \"thanos-querier-96897fc96-xrbqn\" (UID: \"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30\") " pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.472848 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.472823    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhf4t" event={"ID":"8487b755-cece-4303-8108-15623916d5de","Type":"ContainerStarted","Data":"e5d04dc3a696fd6508f1e7d58b30fb56632a2a2f0a55589a3e5fa290db41436b"}
Jul 08 16:36:07.473214 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.472882    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhf4t" event={"ID":"8487b755-cece-4303-8108-15623916d5de","Type":"ContainerStarted","Data":"4846098175db41c161139a1acbd3a3fb67646aa8ea3855860a0bdf0d664dbfa0"}
Jul 08 16:36:07.475078 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.475051    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv" event={"ID":"dfea924a-5429-4fbd-84f1-089e2d2a253c","Type":"ContainerStarted","Data":"6b5e5f14f6691d9e43a4e9bba925e7790fc24670435c80564cd9c07110c0122e"}
Jul 08 16:36:07.483777 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.483738    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42" event={"ID":"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1","Type":"ContainerStarted","Data":"141ac206c92406830b8da91d55303f9f7ef97253d76d16fad92773a8169d5831"}
Jul 08 16:36:07.483844 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.483823    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42" event={"ID":"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1","Type":"ContainerStarted","Data":"28561f827bc2af2f4b034eff232c11dd7f49b4d7c9b300cf60c87cdbbfff47d7"}
Jul 08 16:36:07.483889 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.483849    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42" event={"ID":"8edb418c-0c8e-41f4-b442-a5e6dcefb2e1","Type":"ContainerStarted","Data":"55a447977b534f45154754f099a4463afc8ea16112ceedd2fe4b8d56e7ee2aad"}
Jul 08 16:36:07.503123 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.503053    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bhf4t" podStartSLOduration=2.418938734 podStartE2EDuration="3.503039661s" totalImagesPullingTime="1.000143316s" totalInitContainerRuntime="83.957611ms" isStatefulPod=true podCreationTimestamp="2026-07-08 16:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:07.50133353 +0000 UTC m=+83.037886015" watchObservedRunningTime="2026-07-08 16:36:07.503039661 +0000 UTC m=+83.039592144"
Jul 08 16:36:07.544120 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.544078    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mmxcv" podStartSLOduration=2.038650974 podStartE2EDuration="3.544041487s" totalImagesPullingTime="1.505390513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:07.521745876 +0000 UTC m=+83.058298361" watchObservedRunningTime="2026-07-08 16:36:07.544041487 +0000 UTC m=+83.080593972"
Jul 08 16:36:07.544914 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.544876    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-ljv42" podStartSLOduration=1.947497175 podStartE2EDuration="3.544865771s" totalImagesPullingTime="1.597368596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:07.543407948 +0000 UTC m=+83.079960432" watchObservedRunningTime="2026-07-08 16:36:07.544865771 +0000 UTC m=+83.081418257"
Jul 08 16:36:07.561129 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.561109    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:07.692305 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:07.692272    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-96897fc96-xrbqn"]
Jul 08 16:36:07.695234 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:36:07.695209    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3b3d2c_3fdb_4ed9_985b_142fd9fcfe30.slice/crio-1bb084179f9306a8afab4bbdd5dfc040f0b13bf0ef72e0503e1bb1d2a3072f39 WatchSource:0}: Error finding container 1bb084179f9306a8afab4bbdd5dfc040f0b13bf0ef72e0503e1bb1d2a3072f39: Status 404 returned error can't find the container with id 1bb084179f9306a8afab4bbdd5dfc040f0b13bf0ef72e0503e1bb1d2a3072f39
Jul 08 16:36:08.489602 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:08.489538    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn" event={"ID":"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30","Type":"ContainerStarted","Data":"1bb084179f9306a8afab4bbdd5dfc040f0b13bf0ef72e0503e1bb1d2a3072f39"}
Jul 08 16:36:09.050834 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.050794    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"]
Jul 08 16:36:09.054347 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.054328    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"
Jul 08 16:36:09.058149 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.057960    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 08 16:36:09.058553 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.058518    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-642dl\""
Jul 08 16:36:09.069469 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.069445    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"]
Jul 08 16:36:09.159202 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.159173    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bca384a6-e36b-4da5-86fa-ab002dbb7ae9-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6nk8q\" (UID: \"bca384a6-e36b-4da5-86fa-ab002dbb7ae9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"
Jul 08 16:36:09.159352 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.159232    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlxlp\" (UniqueName: \"kubernetes.io/projected/bca384a6-e36b-4da5-86fa-ab002dbb7ae9-kube-api-access-hlxlp\") pod \"monitoring-plugin-d5dcff7cf-6nk8q\" (UID: \"bca384a6-e36b-4da5-86fa-ab002dbb7ae9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"
Jul 08 16:36:09.259943 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.259905    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bca384a6-e36b-4da5-86fa-ab002dbb7ae9-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6nk8q\" (UID: \"bca384a6-e36b-4da5-86fa-ab002dbb7ae9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"
Jul 08 16:36:09.260090 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.259976    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlxlp\" (UniqueName: \"kubernetes.io/projected/bca384a6-e36b-4da5-86fa-ab002dbb7ae9-kube-api-access-hlxlp\") pod \"monitoring-plugin-d5dcff7cf-6nk8q\" (UID: \"bca384a6-e36b-4da5-86fa-ab002dbb7ae9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"
Jul 08 16:36:09.260090 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:36:09.260070    2550 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 08 16:36:09.260154 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:36:09.260139    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca384a6-e36b-4da5-86fa-ab002dbb7ae9-monitoring-plugin-cert podName:bca384a6-e36b-4da5-86fa-ab002dbb7ae9 nodeName:}" failed. No retries permitted until 2026-07-08 16:36:09.760118157 +0000 UTC m=+85.296670625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/bca384a6-e36b-4da5-86fa-ab002dbb7ae9-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-6nk8q" (UID: "bca384a6-e36b-4da5-86fa-ab002dbb7ae9") : secret "monitoring-plugin-cert" not found
Jul 08 16:36:09.271204 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.271183    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlxlp\" (UniqueName: \"kubernetes.io/projected/bca384a6-e36b-4da5-86fa-ab002dbb7ae9-kube-api-access-hlxlp\") pod \"monitoring-plugin-d5dcff7cf-6nk8q\" (UID: \"bca384a6-e36b-4da5-86fa-ab002dbb7ae9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"
Jul 08 16:36:09.496995 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.496967    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn" event={"ID":"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30","Type":"ContainerStarted","Data":"25e0b1aaba5b77c1186c1f680f9d6bcb60fb0408f497a017f87fba0addf70e68"}
Jul 08 16:36:09.497342 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.497047    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn" event={"ID":"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30","Type":"ContainerStarted","Data":"a0444b5463faad831872238a5c27fc9970569987ee6b6888074a6446f617aa19"}
Jul 08 16:36:09.764041 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:09.763978    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bca384a6-e36b-4da5-86fa-ab002dbb7ae9-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6nk8q\" (UID: \"bca384a6-e36b-4da5-86fa-ab002dbb7ae9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"
Jul 08 16:36:09.764185 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:36:09.764095    2550 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 08 16:36:09.764185 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:36:09.764145    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca384a6-e36b-4da5-86fa-ab002dbb7ae9-monitoring-plugin-cert podName:bca384a6-e36b-4da5-86fa-ab002dbb7ae9 nodeName:}" failed. No retries permitted until 2026-07-08 16:36:10.764132702 +0000 UTC m=+86.300685165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/bca384a6-e36b-4da5-86fa-ab002dbb7ae9-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-6nk8q" (UID: "bca384a6-e36b-4da5-86fa-ab002dbb7ae9") : secret "monitoring-plugin-cert" not found
Jul 08 16:36:10.508026 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:10.507991    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn" event={"ID":"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30","Type":"ContainerStarted","Data":"972d0d32e5901d0191f956c6de17f5f728d711982f73a956ab10e628855c827f"}
Jul 08 16:36:10.773036 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:10.772967    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bca384a6-e36b-4da5-86fa-ab002dbb7ae9-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6nk8q\" (UID: \"bca384a6-e36b-4da5-86fa-ab002dbb7ae9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"
Jul 08 16:36:10.775249 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:10.775224    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bca384a6-e36b-4da5-86fa-ab002dbb7ae9-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6nk8q\" (UID: \"bca384a6-e36b-4da5-86fa-ab002dbb7ae9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"
Jul 08 16:36:10.866320 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:10.866285    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"
Jul 08 16:36:11.004036 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:11.004007    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"]
Jul 08 16:36:11.006758 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:36:11.006733    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca384a6_e36b_4da5_86fa_ab002dbb7ae9.slice/crio-a98b5b64e84154d14cb21fb97e361d4c7f3e10288e35cf4b3707775229cece84 WatchSource:0}: Error finding container a98b5b64e84154d14cb21fb97e361d4c7f3e10288e35cf4b3707775229cece84: Status 404 returned error can't find the container with id a98b5b64e84154d14cb21fb97e361d4c7f3e10288e35cf4b3707775229cece84
Jul 08 16:36:11.513876 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:11.513848    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q" event={"ID":"bca384a6-e36b-4da5-86fa-ab002dbb7ae9","Type":"ContainerStarted","Data":"a98b5b64e84154d14cb21fb97e361d4c7f3e10288e35cf4b3707775229cece84"}
Jul 08 16:36:11.517129 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:11.517102    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn" event={"ID":"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30","Type":"ContainerStarted","Data":"c16f45918208355055c684e661a5895b6cf3a108d7aef716cc076d38b009f3d2"}
Jul 08 16:36:11.517244 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:11.517168    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn" event={"ID":"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30","Type":"ContainerStarted","Data":"c92deb385b63f89d671d40162acd75107b6cf606b8829c4733f8c7b0354964ff"}
Jul 08 16:36:11.517244 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:11.517200    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn" event={"ID":"6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30","Type":"ContainerStarted","Data":"b80a3c0b57834456338f120890bcfb0769699694aee3b42d97a93877a76739cc"}
Jul 08 16:36:11.517357 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:11.517338    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:11.540228 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:11.540188    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn" podStartSLOduration=1.851681517 podStartE2EDuration="4.540176274s" totalImagesPullingTime="2.688494757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:36:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:11.53915528 +0000 UTC m=+87.075707765" watchObservedRunningTime="2026-07-08 16:36:11.540176274 +0000 UTC m=+87.076728759"
Jul 08 16:36:12.522553 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:12.522481    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q" event={"ID":"bca384a6-e36b-4da5-86fa-ab002dbb7ae9","Type":"ContainerStarted","Data":"33674c3524024b48bc0a272818563a43e6ed6844e5119e9f64bc0faa362141db"}
Jul 08 16:36:12.522933 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:12.522848    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"
Jul 08 16:36:12.527225 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:12.527203    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q"
Jul 08 16:36:12.537413 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:12.537361    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nk8q" podStartSLOduration=2.432157621 podStartE2EDuration="3.537343775s" totalImagesPullingTime="1.105186154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:12.536850322 +0000 UTC m=+88.073402810" watchObservedRunningTime="2026-07-08 16:36:12.537343775 +0000 UTC m=+88.073896261"
Jul 08 16:36:12.922454 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:12.922403    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65d7964d55-kbtmv" podUID="59544202-e594-42b7-b1cc-01803b180e5d" containerName="registry" containerID="cri-o://1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e" gracePeriod=30
Jul 08 16:36:13.171768 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.171743    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:36:13.293846 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.293822    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59544202-e594-42b7-b1cc-01803b180e5d-ca-trust-extracted\") pod \"59544202-e594-42b7-b1cc-01803b180e5d\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") "
Jul 08 16:36:13.293971 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.293860    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-bound-sa-token\") pod \"59544202-e594-42b7-b1cc-01803b180e5d\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") "
Jul 08 16:36:13.293971 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.293883    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59544202-e594-42b7-b1cc-01803b180e5d-registry-certificates\") pod \"59544202-e594-42b7-b1cc-01803b180e5d\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") "
Jul 08 16:36:13.293971 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.293921    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59544202-e594-42b7-b1cc-01803b180e5d-image-registry-private-configuration\") pod \"59544202-e594-42b7-b1cc-01803b180e5d\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") "
Jul 08 16:36:13.293971 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.293937    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls\") pod \"59544202-e594-42b7-b1cc-01803b180e5d\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") "
Jul 08 16:36:13.293971 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.293953    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59544202-e594-42b7-b1cc-01803b180e5d-installation-pull-secrets\") pod \"59544202-e594-42b7-b1cc-01803b180e5d\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") "
Jul 08 16:36:13.294178 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.293974    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59544202-e594-42b7-b1cc-01803b180e5d-trusted-ca\") pod \"59544202-e594-42b7-b1cc-01803b180e5d\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") "
Jul 08 16:36:13.294178 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.294006    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llvxj\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-kube-api-access-llvxj\") pod \"59544202-e594-42b7-b1cc-01803b180e5d\" (UID: \"59544202-e594-42b7-b1cc-01803b180e5d\") "
Jul 08 16:36:13.294537 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.294476    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59544202-e594-42b7-b1cc-01803b180e5d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "59544202-e594-42b7-b1cc-01803b180e5d" (UID: "59544202-e594-42b7-b1cc-01803b180e5d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:13.294774 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.294745    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59544202-e594-42b7-b1cc-01803b180e5d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "59544202-e594-42b7-b1cc-01803b180e5d" (UID: "59544202-e594-42b7-b1cc-01803b180e5d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:13.296672 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.296640    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "59544202-e594-42b7-b1cc-01803b180e5d" (UID: "59544202-e594-42b7-b1cc-01803b180e5d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:13.296775 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.296668    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59544202-e594-42b7-b1cc-01803b180e5d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "59544202-e594-42b7-b1cc-01803b180e5d" (UID: "59544202-e594-42b7-b1cc-01803b180e5d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:13.296775 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.296729    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-kube-api-access-llvxj" (OuterVolumeSpecName: "kube-api-access-llvxj") pod "59544202-e594-42b7-b1cc-01803b180e5d" (UID: "59544202-e594-42b7-b1cc-01803b180e5d"). InnerVolumeSpecName "kube-api-access-llvxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:13.296775 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.296758    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59544202-e594-42b7-b1cc-01803b180e5d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "59544202-e594-42b7-b1cc-01803b180e5d" (UID: "59544202-e594-42b7-b1cc-01803b180e5d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:13.296967 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.296803    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "59544202-e594-42b7-b1cc-01803b180e5d" (UID: "59544202-e594-42b7-b1cc-01803b180e5d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:13.303125 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.303102    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59544202-e594-42b7-b1cc-01803b180e5d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "59544202-e594-42b7-b1cc-01803b180e5d" (UID: "59544202-e594-42b7-b1cc-01803b180e5d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:36:13.394604 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.394582    2550 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59544202-e594-42b7-b1cc-01803b180e5d-ca-trust-extracted\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:13.394604 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.394603    2550 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-bound-sa-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:13.394717 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.394611    2550 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59544202-e594-42b7-b1cc-01803b180e5d-registry-certificates\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:13.394717 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.394620    2550 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59544202-e594-42b7-b1cc-01803b180e5d-image-registry-private-configuration\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:13.394717 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.394629    2550 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-registry-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:13.394717 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.394636    2550 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59544202-e594-42b7-b1cc-01803b180e5d-installation-pull-secrets\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:13.394717 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.394644    2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59544202-e594-42b7-b1cc-01803b180e5d-trusted-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:13.394717 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.394651    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llvxj\" (UniqueName: \"kubernetes.io/projected/59544202-e594-42b7-b1cc-01803b180e5d-kube-api-access-llvxj\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:13.527198 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.527167    2550 generic.go:356] "Generic (PLEG): container finished" podID="59544202-e594-42b7-b1cc-01803b180e5d" containerID="1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e" exitCode=0
Jul 08 16:36:13.527458 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.527214    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d7964d55-kbtmv"
Jul 08 16:36:13.527458 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.527246    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d7964d55-kbtmv" event={"ID":"59544202-e594-42b7-b1cc-01803b180e5d","Type":"ContainerDied","Data":"1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e"}
Jul 08 16:36:13.527458 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.527299    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d7964d55-kbtmv" event={"ID":"59544202-e594-42b7-b1cc-01803b180e5d","Type":"ContainerDied","Data":"416e33d3e4781a4be6224230763e54022c860a224359a4a18c85561b51fb7123"}
Jul 08 16:36:13.527458 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.527323    2550 scope.go:117] "RemoveContainer" containerID="1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e"
Jul 08 16:36:13.535651 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.535538    2550 scope.go:117] "RemoveContainer" containerID="1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e"
Jul 08 16:36:13.535896 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:36:13.535868    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e\": container with ID starting with 1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e not found: ID does not exist" containerID="1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e"
Jul 08 16:36:13.535954 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.535904    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e"} err="failed to get container status \"1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e\": rpc error: code = NotFound desc = could not find container \"1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e\": container with ID starting with 1e16a25802cc7ce6b54ac4cc6797a5fb837d2eaa3f9f272238b694bcc5abfc6e not found: ID does not exist"
Jul 08 16:36:13.548350 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.548303    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65d7964d55-kbtmv"]
Jul 08 16:36:13.552160 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:13.552142    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65d7964d55-kbtmv"]
Jul 08 16:36:15.028137 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:15.028109    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59544202-e594-42b7-b1cc-01803b180e5d" path="/var/lib/kubelet/pods/59544202-e594-42b7-b1cc-01803b180e5d/volumes"
Jul 08 16:36:17.528293 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:17.528267    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-96897fc96-xrbqn"
Jul 08 16:36:20.266269 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:20.266240    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8468f948d5-x6qg9"]
Jul 08 16:36:29.419880 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:29.419853    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-79p9f"
Jul 08 16:36:45.285960 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.285904    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-8468f948d5-x6qg9" podUID="6b23c066-a365-4e6e-8170-93e0e2bc999d" containerName="console" containerID="cri-o://d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118" gracePeriod=15
Jul 08 16:36:45.513890 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.513869    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:36:45.612494 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.612422    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-serving-cert\") pod \"6b23c066-a365-4e6e-8170-93e0e2bc999d\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") "
Jul 08 16:36:45.612494 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.612461    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-oauth-config\") pod \"6b23c066-a365-4e6e-8170-93e0e2bc999d\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") "
Jul 08 16:36:45.612494 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.612488    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrnnm\" (UniqueName: \"kubernetes.io/projected/6b23c066-a365-4e6e-8170-93e0e2bc999d-kube-api-access-jrnnm\") pod \"6b23c066-a365-4e6e-8170-93e0e2bc999d\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") "
Jul 08 16:36:45.612737 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.612516    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-oauth-serving-cert\") pod \"6b23c066-a365-4e6e-8170-93e0e2bc999d\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") "
Jul 08 16:36:45.612737 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.612551    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-service-ca\") pod \"6b23c066-a365-4e6e-8170-93e0e2bc999d\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") "
Jul 08 16:36:45.612737 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.612621    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-config\") pod \"6b23c066-a365-4e6e-8170-93e0e2bc999d\" (UID: \"6b23c066-a365-4e6e-8170-93e0e2bc999d\") "
Jul 08 16:36:45.613030 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.613003    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6b23c066-a365-4e6e-8170-93e0e2bc999d" (UID: "6b23c066-a365-4e6e-8170-93e0e2bc999d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:45.613030 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.613015    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-service-ca" (OuterVolumeSpecName: "service-ca") pod "6b23c066-a365-4e6e-8170-93e0e2bc999d" (UID: "6b23c066-a365-4e6e-8170-93e0e2bc999d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:45.613133 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.613048    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-config" (OuterVolumeSpecName: "console-config") pod "6b23c066-a365-4e6e-8170-93e0e2bc999d" (UID: "6b23c066-a365-4e6e-8170-93e0e2bc999d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:45.614930 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.614907    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6b23c066-a365-4e6e-8170-93e0e2bc999d" (UID: "6b23c066-a365-4e6e-8170-93e0e2bc999d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:45.615115 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.615091    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6b23c066-a365-4e6e-8170-93e0e2bc999d" (UID: "6b23c066-a365-4e6e-8170-93e0e2bc999d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:45.615192 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.615174    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b23c066-a365-4e6e-8170-93e0e2bc999d-kube-api-access-jrnnm" (OuterVolumeSpecName: "kube-api-access-jrnnm") pod "6b23c066-a365-4e6e-8170-93e0e2bc999d" (UID: "6b23c066-a365-4e6e-8170-93e0e2bc999d"). InnerVolumeSpecName "kube-api-access-jrnnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:45.646005 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.645974    2550 generic.go:356] "Generic (PLEG): container finished" podID="6b23c066-a365-4e6e-8170-93e0e2bc999d" containerID="d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118" exitCode=0
Jul 08 16:36:45.646106 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.646028    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8468f948d5-x6qg9"
Jul 08 16:36:45.646106 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.646069    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8468f948d5-x6qg9" event={"ID":"6b23c066-a365-4e6e-8170-93e0e2bc999d","Type":"ContainerDied","Data":"d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118"}
Jul 08 16:36:45.646183 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.646120    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8468f948d5-x6qg9" event={"ID":"6b23c066-a365-4e6e-8170-93e0e2bc999d","Type":"ContainerDied","Data":"30c17d16c360578eb75cdfc26fe91bae6a3c98d5db94ea89778b9869cb7820a2"}
Jul 08 16:36:45.646183 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.646147    2550 scope.go:117] "RemoveContainer" containerID="d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118"
Jul 08 16:36:45.654052 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.654038    2550 scope.go:117] "RemoveContainer" containerID="d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118"
Jul 08 16:36:45.654306 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:36:45.654287    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118\": container with ID starting with d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118 not found: ID does not exist" containerID="d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118"
Jul 08 16:36:45.654347 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.654314    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118"} err="failed to get container status \"d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118\": rpc error: code = NotFound desc = could not find container \"d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118\": container with ID starting with d3603bbe4824aa3f981fa6b40c8c3417e9ef5eb3bc7b205dbd37a42c1057a118 not found: ID does not exist"
Jul 08 16:36:45.666902 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.666882    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8468f948d5-x6qg9"]
Jul 08 16:36:45.670458 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.670440    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8468f948d5-x6qg9"]
Jul 08 16:36:45.713787 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.713769    2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-serving-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:45.713856 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.713788    2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-oauth-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:45.713856 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.713799    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrnnm\" (UniqueName: \"kubernetes.io/projected/6b23c066-a365-4e6e-8170-93e0e2bc999d-kube-api-access-jrnnm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:45.713856 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.713810    2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-oauth-serving-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:45.713856 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.713824    2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:45.713856 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:45.713832    2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b23c066-a365-4e6e-8170-93e0e2bc999d-console-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:36:47.027087 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:36:47.027057    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b23c066-a365-4e6e-8170-93e0e2bc999d" path="/var/lib/kubelet/pods/6b23c066-a365-4e6e-8170-93e0e2bc999d/volumes"
Jul 08 16:37:34.460688 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.460657    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-584fff874f-gvqz2"]
Jul 08 16:37:34.461088 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.460913    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59544202-e594-42b7-b1cc-01803b180e5d" containerName="registry"
Jul 08 16:37:34.461088 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.460923    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="59544202-e594-42b7-b1cc-01803b180e5d" containerName="registry"
Jul 08 16:37:34.461088 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.460931    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b23c066-a365-4e6e-8170-93e0e2bc999d" containerName="console"
Jul 08 16:37:34.461088 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.460936    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b23c066-a365-4e6e-8170-93e0e2bc999d" containerName="console"
Jul 08 16:37:34.461088 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.460977    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b23c066-a365-4e6e-8170-93e0e2bc999d" containerName="console"
Jul 08 16:37:34.461088 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.460984    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="59544202-e594-42b7-b1cc-01803b180e5d" containerName="registry"
Jul 08 16:37:34.463594 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.463554    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.466447 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.466414    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fpsds\""
Jul 08 16:37:34.466698 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.466683    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 16:37:34.467497 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.467476    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 16:37:34.467619 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.467591    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 16:37:34.467789 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.467770    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 16:37:34.467839 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.467826    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 16:37:34.467839 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.467831    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 16:37:34.468440 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.468425    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 16:37:34.472372 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.472354    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 08 16:37:34.476299 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.476281    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-584fff874f-gvqz2"]
Jul 08 16:37:34.531077 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.531057    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/513903c2-f218-4bf9-8823-d213ddb20d22-console-serving-cert\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.531180 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.531084    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-console-config\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.531180 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.531111    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7c2\" (UniqueName: \"kubernetes.io/projected/513903c2-f218-4bf9-8823-d213ddb20d22-kube-api-access-kn7c2\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.531180 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.531132    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-oauth-serving-cert\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.531341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.531215    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/513903c2-f218-4bf9-8823-d213ddb20d22-console-oauth-config\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.531341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.531265    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-trusted-ca-bundle\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.531341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.531295    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-service-ca\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.632127 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.632101    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/513903c2-f218-4bf9-8823-d213ddb20d22-console-serving-cert\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.632266 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.632132    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-console-config\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.632266 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.632158    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7c2\" (UniqueName: \"kubernetes.io/projected/513903c2-f218-4bf9-8823-d213ddb20d22-kube-api-access-kn7c2\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.632266 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.632178    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-oauth-serving-cert\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.632266 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.632200    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/513903c2-f218-4bf9-8823-d213ddb20d22-console-oauth-config\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.632266 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.632241    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-trusted-ca-bundle\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.632449 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.632269    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-service-ca\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.633154 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.633130    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-oauth-serving-cert\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.633263 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.633137    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-service-ca\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.633263 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.633199    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-trusted-ca-bundle\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.633406 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.633384    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-console-config\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.635206 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.635168    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/513903c2-f218-4bf9-8823-d213ddb20d22-console-serving-cert\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.635786 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.635763    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/513903c2-f218-4bf9-8823-d213ddb20d22-console-oauth-config\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.640222 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.640201    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7c2\" (UniqueName: \"kubernetes.io/projected/513903c2-f218-4bf9-8823-d213ddb20d22-kube-api-access-kn7c2\") pod \"console-584fff874f-gvqz2\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") " pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.772667 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.772603    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:34.921001 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:34.920862    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-584fff874f-gvqz2"]
Jul 08 16:37:34.923500 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:37:34.923471    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513903c2_f218_4bf9_8823_d213ddb20d22.slice/crio-fa9e9c6df33c415e4496e979d215f2c3575fb177b0df921cb79d1570d99348f5 WatchSource:0}: Error finding container fa9e9c6df33c415e4496e979d215f2c3575fb177b0df921cb79d1570d99348f5: Status 404 returned error can't find the container with id fa9e9c6df33c415e4496e979d215f2c3575fb177b0df921cb79d1570d99348f5
Jul 08 16:37:35.819532 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:35.819456    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584fff874f-gvqz2" event={"ID":"513903c2-f218-4bf9-8823-d213ddb20d22","Type":"ContainerStarted","Data":"042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a"}
Jul 08 16:37:35.819927 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:35.819531    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584fff874f-gvqz2" event={"ID":"513903c2-f218-4bf9-8823-d213ddb20d22","Type":"ContainerStarted","Data":"fa9e9c6df33c415e4496e979d215f2c3575fb177b0df921cb79d1570d99348f5"}
Jul 08 16:37:35.839017 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:35.838965    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-584fff874f-gvqz2" podStartSLOduration=1.838951532 podStartE2EDuration="1.838951532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:37:35.837531952 +0000 UTC m=+171.374084437" watchObservedRunningTime="2026-07-08 16:37:35.838951532 +0000 UTC m=+171.375504019"
Jul 08 16:37:41.302626 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.302596    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-584fff874f-gvqz2"]
Jul 08 16:37:41.368499 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.368466    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76449d99d9-sf492"]
Jul 08 16:37:41.371784 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.371768    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.381878 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.381859    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76449d99d9-sf492"]
Jul 08 16:37:41.478507 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.478481    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdnlp\" (UniqueName: \"kubernetes.io/projected/335966fe-5da9-4946-85da-4040a5765c89-kube-api-access-gdnlp\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.478659 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.478519    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/335966fe-5da9-4946-85da-4040a5765c89-console-oauth-config\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.478659 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.478540    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/335966fe-5da9-4946-85da-4040a5765c89-service-ca\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.478659 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.478629    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/335966fe-5da9-4946-85da-4040a5765c89-console-config\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.478762 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.478698    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/335966fe-5da9-4946-85da-4040a5765c89-console-serving-cert\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.478762 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.478722    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/335966fe-5da9-4946-85da-4040a5765c89-oauth-serving-cert\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.478762 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.478737    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/335966fe-5da9-4946-85da-4040a5765c89-trusted-ca-bundle\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.579905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.579809    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/335966fe-5da9-4946-85da-4040a5765c89-console-serving-cert\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.579905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.579859    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/335966fe-5da9-4946-85da-4040a5765c89-oauth-serving-cert\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.579905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.579876    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/335966fe-5da9-4946-85da-4040a5765c89-trusted-ca-bundle\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.579905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.579913    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdnlp\" (UniqueName: \"kubernetes.io/projected/335966fe-5da9-4946-85da-4040a5765c89-kube-api-access-gdnlp\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.580171 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.579946    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/335966fe-5da9-4946-85da-4040a5765c89-console-oauth-config\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.580171 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.579979    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/335966fe-5da9-4946-85da-4040a5765c89-service-ca\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.580171 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.580008    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/335966fe-5da9-4946-85da-4040a5765c89-console-config\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.580713 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.580684    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/335966fe-5da9-4946-85da-4040a5765c89-oauth-serving-cert\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.580810 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.580749    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/335966fe-5da9-4946-85da-4040a5765c89-console-config\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.580810 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.580771    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/335966fe-5da9-4946-85da-4040a5765c89-service-ca\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.581103 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.581082    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/335966fe-5da9-4946-85da-4040a5765c89-trusted-ca-bundle\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.582391 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.582370    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/335966fe-5da9-4946-85da-4040a5765c89-console-oauth-config\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.582468 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.582450    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/335966fe-5da9-4946-85da-4040a5765c89-console-serving-cert\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.588041 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.588021    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdnlp\" (UniqueName: \"kubernetes.io/projected/335966fe-5da9-4946-85da-4040a5765c89-kube-api-access-gdnlp\") pod \"console-76449d99d9-sf492\" (UID: \"335966fe-5da9-4946-85da-4040a5765c89\") " pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.680750 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.680725    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:41.817802 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.817775    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76449d99d9-sf492"]
Jul 08 16:37:41.821392 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:37:41.821366    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335966fe_5da9_4946_85da_4040a5765c89.slice/crio-f3f8fbc08919ae818c59515b8a0c537cd48e93a5b21b643e03eb77e124999f4b WatchSource:0}: Error finding container f3f8fbc08919ae818c59515b8a0c537cd48e93a5b21b643e03eb77e124999f4b: Status 404 returned error can't find the container with id f3f8fbc08919ae818c59515b8a0c537cd48e93a5b21b643e03eb77e124999f4b
Jul 08 16:37:41.840407 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:41.840374    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76449d99d9-sf492" event={"ID":"335966fe-5da9-4946-85da-4040a5765c89","Type":"ContainerStarted","Data":"f3f8fbc08919ae818c59515b8a0c537cd48e93a5b21b643e03eb77e124999f4b"}
Jul 08 16:37:42.845526 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:42.845495    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76449d99d9-sf492" event={"ID":"335966fe-5da9-4946-85da-4040a5765c89","Type":"ContainerStarted","Data":"cfb2b22285597e24dac5666c9d317793d78678bf70d2eec14f83211f6311c9a3"}
Jul 08 16:37:42.867717 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:42.867678    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76449d99d9-sf492" podStartSLOduration=1.867663515 podStartE2EDuration="1.867663515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:37:42.865781625 +0000 UTC m=+178.402334111" watchObservedRunningTime="2026-07-08 16:37:42.867663515 +0000 UTC m=+178.404216007"
Jul 08 16:37:44.773213 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:44.773185    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:37:51.681240 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:51.681202    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:51.681591 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:51.681251    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:51.685528 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:51.685508    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:37:51.877886 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:37:51.877867    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76449d99d9-sf492"
Jul 08 16:38:06.325525 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.325485    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-584fff874f-gvqz2" podUID="513903c2-f218-4bf9-8823-d213ddb20d22" containerName="console" containerID="cri-o://042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a" gracePeriod=15
Jul 08 16:38:06.557088 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.557063    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:38:06.651643 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.651593    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-service-ca\") pod \"513903c2-f218-4bf9-8823-d213ddb20d22\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") "
Jul 08 16:38:06.651643 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.651622    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-trusted-ca-bundle\") pod \"513903c2-f218-4bf9-8823-d213ddb20d22\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") "
Jul 08 16:38:06.651808 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.651652    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn7c2\" (UniqueName: \"kubernetes.io/projected/513903c2-f218-4bf9-8823-d213ddb20d22-kube-api-access-kn7c2\") pod \"513903c2-f218-4bf9-8823-d213ddb20d22\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") "
Jul 08 16:38:06.651808 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.651678    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/513903c2-f218-4bf9-8823-d213ddb20d22-console-serving-cert\") pod \"513903c2-f218-4bf9-8823-d213ddb20d22\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") "
Jul 08 16:38:06.651808 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.651693    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/513903c2-f218-4bf9-8823-d213ddb20d22-console-oauth-config\") pod \"513903c2-f218-4bf9-8823-d213ddb20d22\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") "
Jul 08 16:38:06.651808 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.651712    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-console-config\") pod \"513903c2-f218-4bf9-8823-d213ddb20d22\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") "
Jul 08 16:38:06.651808 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.651744    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-oauth-serving-cert\") pod \"513903c2-f218-4bf9-8823-d213ddb20d22\" (UID: \"513903c2-f218-4bf9-8823-d213ddb20d22\") "
Jul 08 16:38:06.652122 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.652094    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-service-ca" (OuterVolumeSpecName: "service-ca") pod "513903c2-f218-4bf9-8823-d213ddb20d22" (UID: "513903c2-f218-4bf9-8823-d213ddb20d22"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:38:06.652191 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.652130    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "513903c2-f218-4bf9-8823-d213ddb20d22" (UID: "513903c2-f218-4bf9-8823-d213ddb20d22"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:38:06.652236 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.652186    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "513903c2-f218-4bf9-8823-d213ddb20d22" (UID: "513903c2-f218-4bf9-8823-d213ddb20d22"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:38:06.652236 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.652184    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-console-config" (OuterVolumeSpecName: "console-config") pod "513903c2-f218-4bf9-8823-d213ddb20d22" (UID: "513903c2-f218-4bf9-8823-d213ddb20d22"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:38:06.653877 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.653856    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513903c2-f218-4bf9-8823-d213ddb20d22-kube-api-access-kn7c2" (OuterVolumeSpecName: "kube-api-access-kn7c2") pod "513903c2-f218-4bf9-8823-d213ddb20d22" (UID: "513903c2-f218-4bf9-8823-d213ddb20d22"). InnerVolumeSpecName "kube-api-access-kn7c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:38:06.654262 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.654244    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513903c2-f218-4bf9-8823-d213ddb20d22-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "513903c2-f218-4bf9-8823-d213ddb20d22" (UID: "513903c2-f218-4bf9-8823-d213ddb20d22"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:38:06.654337 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.654264    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513903c2-f218-4bf9-8823-d213ddb20d22-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "513903c2-f218-4bf9-8823-d213ddb20d22" (UID: "513903c2-f218-4bf9-8823-d213ddb20d22"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:38:06.753117 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.753095    2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/513903c2-f218-4bf9-8823-d213ddb20d22-console-serving-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:38:06.753117 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.753113    2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/513903c2-f218-4bf9-8823-d213ddb20d22-console-oauth-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:38:06.753222 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.753122    2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-console-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:38:06.753222 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.753131    2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-oauth-serving-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:38:06.753222 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.753138    2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:38:06.753222 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.753146    2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/513903c2-f218-4bf9-8823-d213ddb20d22-trusted-ca-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:38:06.753222 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.753153    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn7c2\" (UniqueName: \"kubernetes.io/projected/513903c2-f218-4bf9-8823-d213ddb20d22-kube-api-access-kn7c2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:38:06.927036 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.926975    2550 generic.go:356] "Generic (PLEG): container finished" podID="513903c2-f218-4bf9-8823-d213ddb20d22" containerID="042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a" exitCode=0
Jul 08 16:38:06.927036 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.927016    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584fff874f-gvqz2" event={"ID":"513903c2-f218-4bf9-8823-d213ddb20d22","Type":"ContainerDied","Data":"042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a"}
Jul 08 16:38:06.927187 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.927072    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584fff874f-gvqz2"
Jul 08 16:38:06.927187 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.927084    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584fff874f-gvqz2" event={"ID":"513903c2-f218-4bf9-8823-d213ddb20d22","Type":"ContainerDied","Data":"fa9e9c6df33c415e4496e979d215f2c3575fb177b0df921cb79d1570d99348f5"}
Jul 08 16:38:06.927187 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.927105    2550 scope.go:117] "RemoveContainer" containerID="042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a"
Jul 08 16:38:06.935312 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.935298    2550 scope.go:117] "RemoveContainer" containerID="042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a"
Jul 08 16:38:06.935553 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:38:06.935536    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a\": container with ID starting with 042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a not found: ID does not exist" containerID="042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a"
Jul 08 16:38:06.935673 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.935582    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a"} err="failed to get container status \"042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a\": rpc error: code = NotFound desc = could not find container \"042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a\": container with ID starting with 042225e4d053fcc1c6ef68650271c542be38f5d2b192df8c8fc2c230acdb2b8a not found: ID does not exist"
Jul 08 16:38:06.947987 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.947965    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-584fff874f-gvqz2"]
Jul 08 16:38:06.951101 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:06.951082    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-584fff874f-gvqz2"]
Jul 08 16:38:07.026545 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:07.026521    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="513903c2-f218-4bf9-8823-d213ddb20d22" path="/var/lib/kubelet/pods/513903c2-f218-4bf9-8823-d213ddb20d22/volumes"
Jul 08 16:38:42.205344 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.205295    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92"]
Jul 08 16:38:42.205940 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.205922    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513903c2-f218-4bf9-8823-d213ddb20d22" containerName="console"
Jul 08 16:38:42.206016 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.205945    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="513903c2-f218-4bf9-8823-d213ddb20d22" containerName="console"
Jul 08 16:38:42.206067 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.206013    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="513903c2-f218-4bf9-8823-d213ddb20d22" containerName="console"
Jul 08 16:38:42.209010 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.208991    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92"
Jul 08 16:38:42.211332 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.211313    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-g25vs\""
Jul 08 16:38:42.212214 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.212195    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 08 16:38:42.212314 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.212195    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 08 16:38:42.212455 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.212441    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 08 16:38:42.212512 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.212472    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 08 16:38:42.218909 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.218888    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92"]
Jul 08 16:38:42.255805 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.255786    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"]
Jul 08 16:38:42.260920 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.260898    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.264040 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.264017    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 08 16:38:42.264122 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.264041    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 08 16:38:42.264162 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.264122    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 08 16:38:42.264377 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.264361    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 08 16:38:42.275279 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.275260    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"]
Jul 08 16:38:42.389816 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.389789    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.389936 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.389822    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vblps\" (UniqueName: \"kubernetes.io/projected/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-kube-api-access-vblps\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.389936 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.389860    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.389936 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.389879    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.389936 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.389911    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a775ddba-6a29-49b6-a507-96dcdfddc7cf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6ddf57c658-cvk92\" (UID: \"a775ddba-6a29-49b6-a507-96dcdfddc7cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92"
Jul 08 16:38:42.390081 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.389936    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qb6k\" (UniqueName: \"kubernetes.io/projected/a775ddba-6a29-49b6-a507-96dcdfddc7cf-kube-api-access-2qb6k\") pod \"managed-serviceaccount-addon-agent-6ddf57c658-cvk92\" (UID: \"a775ddba-6a29-49b6-a507-96dcdfddc7cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92"
Jul 08 16:38:42.390081 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.389959    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-hub\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.390081 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.389996    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-ca\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.490800 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.490711    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.490800 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.490750    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.490800 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.490766    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a775ddba-6a29-49b6-a507-96dcdfddc7cf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6ddf57c658-cvk92\" (UID: \"a775ddba-6a29-49b6-a507-96dcdfddc7cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92"
Jul 08 16:38:42.490800 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.490793    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qb6k\" (UniqueName: \"kubernetes.io/projected/a775ddba-6a29-49b6-a507-96dcdfddc7cf-kube-api-access-2qb6k\") pod \"managed-serviceaccount-addon-agent-6ddf57c658-cvk92\" (UID: \"a775ddba-6a29-49b6-a507-96dcdfddc7cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92"
Jul 08 16:38:42.491108 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.490812    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-hub\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.491108 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.490842    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-ca\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.491108 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.490867    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.491108 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.490901    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vblps\" (UniqueName: \"kubernetes.io/projected/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-kube-api-access-vblps\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.491798 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.491588    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.493427 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.493404    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.493530 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.493515    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-hub\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.493682 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.493667    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-ca\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.493726 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.493675    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.493812 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.493797    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a775ddba-6a29-49b6-a507-96dcdfddc7cf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6ddf57c658-cvk92\" (UID: \"a775ddba-6a29-49b6-a507-96dcdfddc7cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92"
Jul 08 16:38:42.498759 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.498732    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vblps\" (UniqueName: \"kubernetes.io/projected/d254dbab-01d9-4f4b-b202-b3549b3bbd6a-kube-api-access-vblps\") pod \"cluster-proxy-proxy-agent-774b8d4878-7pdvb\" (UID: \"d254dbab-01d9-4f4b-b202-b3549b3bbd6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.499095 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.499079    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qb6k\" (UniqueName: \"kubernetes.io/projected/a775ddba-6a29-49b6-a507-96dcdfddc7cf-kube-api-access-2qb6k\") pod \"managed-serviceaccount-addon-agent-6ddf57c658-cvk92\" (UID: \"a775ddba-6a29-49b6-a507-96dcdfddc7cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92"
Jul 08 16:38:42.533057 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.533034    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92"
Jul 08 16:38:42.570122 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.570094    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"
Jul 08 16:38:42.666769 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.666454    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92"]
Jul 08 16:38:42.670401 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:38:42.670346    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda775ddba_6a29_49b6_a507_96dcdfddc7cf.slice/crio-5124317b848a4f091337217c6354c1bf778feec837c449f3a6038cfd7f345bf7 WatchSource:0}: Error finding container 5124317b848a4f091337217c6354c1bf778feec837c449f3a6038cfd7f345bf7: Status 404 returned error can't find the container with id 5124317b848a4f091337217c6354c1bf778feec837c449f3a6038cfd7f345bf7
Jul 08 16:38:42.712056 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:42.712029    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb"]
Jul 08 16:38:42.714640 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:38:42.714616    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd254dbab_01d9_4f4b_b202_b3549b3bbd6a.slice/crio-a1aec1dbfa0e62e7a29c2a0e6341ffa9fa0fa0e457ed536b59dc19461aa3050e WatchSource:0}: Error finding container a1aec1dbfa0e62e7a29c2a0e6341ffa9fa0fa0e457ed536b59dc19461aa3050e: Status 404 returned error can't find the container with id a1aec1dbfa0e62e7a29c2a0e6341ffa9fa0fa0e457ed536b59dc19461aa3050e
Jul 08 16:38:43.047988 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:43.047956    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb" event={"ID":"d254dbab-01d9-4f4b-b202-b3549b3bbd6a","Type":"ContainerStarted","Data":"a1aec1dbfa0e62e7a29c2a0e6341ffa9fa0fa0e457ed536b59dc19461aa3050e"}
Jul 08 16:38:43.048989 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:43.048968    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92" event={"ID":"a775ddba-6a29-49b6-a507-96dcdfddc7cf","Type":"ContainerStarted","Data":"5124317b848a4f091337217c6354c1bf778feec837c449f3a6038cfd7f345bf7"}
Jul 08 16:38:47.068140 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:47.068100    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb" event={"ID":"d254dbab-01d9-4f4b-b202-b3549b3bbd6a","Type":"ContainerStarted","Data":"90e2ee1e93bc26e81ab8c9de052c741e26afe462dfc34bfad976e089e27597dd"}
Jul 08 16:38:47.068140 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:47.068142    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb" event={"ID":"d254dbab-01d9-4f4b-b202-b3549b3bbd6a","Type":"ContainerStarted","Data":"38ef0697ab7031a2980a1142857aa7eba67a7afbf0e1031ba6d86572c2f5148c"}
Jul 08 16:38:47.068649 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:47.068164    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb" event={"ID":"d254dbab-01d9-4f4b-b202-b3549b3bbd6a","Type":"ContainerStarted","Data":"48cbabe03e2d49f4192f8d0cbec8d72b7360e1b76bd667567bff0cc727ad0982"}
Jul 08 16:38:47.069589 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:47.069544    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92" event={"ID":"a775ddba-6a29-49b6-a507-96dcdfddc7cf","Type":"ContainerStarted","Data":"28e3b1f5e3a650025f379ff7990c21f7197e7f284370a0c908d6d26a541bd641"}
Jul 08 16:38:47.090115 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:47.090070    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b8d4878-7pdvb" podStartSLOduration=1.233659985 podStartE2EDuration="5.090035231s" totalImagesPullingTime="3.856375246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:38:47.08811007 +0000 UTC m=+242.624662554" watchObservedRunningTime="2026-07-08 16:38:47.090035231 +0000 UTC m=+242.626587715"
Jul 08 16:38:47.105329 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:38:47.105289    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ddf57c658-cvk92" podStartSLOduration=1.262638087 podStartE2EDuration="5.105277132s" totalImagesPullingTime="3.842639045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:38:47.104993001 +0000 UTC m=+242.641545485" watchObservedRunningTime="2026-07-08 16:38:47.105277132 +0000 UTC m=+242.641829616"
Jul 08 16:39:44.823801 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:39:44.823764    2550 kubelet.go:1648] "Image garbage collection succeeded"
Jul 08 16:40:23.260213 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.260185    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-qxlf9"]
Jul 08 16:40:23.262506 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.262491    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qxlf9"
Jul 08 16:40:23.265480 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.265460    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 08 16:40:23.266110 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.266089    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-pcmrj\""
Jul 08 16:40:23.266329 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.266309    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 08 16:40:23.266432 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.266350    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 08 16:40:23.272158 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.272140    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qxlf9"]
Jul 08 16:40:23.355290 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.355266    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b69b200d-32dc-4711-9127-b7a62b85a04e-data\") pod \"seaweedfs-64568bcd49-qxlf9\" (UID: \"b69b200d-32dc-4711-9127-b7a62b85a04e\") " pod="kserve/seaweedfs-64568bcd49-qxlf9"
Jul 08 16:40:23.355374 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.355323    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7cks\" (UniqueName: \"kubernetes.io/projected/b69b200d-32dc-4711-9127-b7a62b85a04e-kube-api-access-l7cks\") pod \"seaweedfs-64568bcd49-qxlf9\" (UID: \"b69b200d-32dc-4711-9127-b7a62b85a04e\") " pod="kserve/seaweedfs-64568bcd49-qxlf9"
Jul 08 16:40:23.455690 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.455635    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b69b200d-32dc-4711-9127-b7a62b85a04e-data\") pod \"seaweedfs-64568bcd49-qxlf9\" (UID: \"b69b200d-32dc-4711-9127-b7a62b85a04e\") " pod="kserve/seaweedfs-64568bcd49-qxlf9"
Jul 08 16:40:23.455773 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.455699    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7cks\" (UniqueName: \"kubernetes.io/projected/b69b200d-32dc-4711-9127-b7a62b85a04e-kube-api-access-l7cks\") pod \"seaweedfs-64568bcd49-qxlf9\" (UID: \"b69b200d-32dc-4711-9127-b7a62b85a04e\") " pod="kserve/seaweedfs-64568bcd49-qxlf9"
Jul 08 16:40:23.455975 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.455961    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b69b200d-32dc-4711-9127-b7a62b85a04e-data\") pod \"seaweedfs-64568bcd49-qxlf9\" (UID: \"b69b200d-32dc-4711-9127-b7a62b85a04e\") " pod="kserve/seaweedfs-64568bcd49-qxlf9"
Jul 08 16:40:23.464544 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.464524    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7cks\" (UniqueName: \"kubernetes.io/projected/b69b200d-32dc-4711-9127-b7a62b85a04e-kube-api-access-l7cks\") pod \"seaweedfs-64568bcd49-qxlf9\" (UID: \"b69b200d-32dc-4711-9127-b7a62b85a04e\") " pod="kserve/seaweedfs-64568bcd49-qxlf9"
Jul 08 16:40:23.571680 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.571659    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qxlf9"
Jul 08 16:40:23.690651 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.690465    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qxlf9"]
Jul 08 16:40:23.693347 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:40:23.693321    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69b200d_32dc_4711_9127_b7a62b85a04e.slice/crio-2dc2c333e41458e28605c2c666d43a96585802b9f71566ece7a6a8288ed447c4 WatchSource:0}: Error finding container 2dc2c333e41458e28605c2c666d43a96585802b9f71566ece7a6a8288ed447c4: Status 404 returned error can't find the container with id 2dc2c333e41458e28605c2c666d43a96585802b9f71566ece7a6a8288ed447c4
Jul 08 16:40:23.694796 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:23.694763    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:40:24.399005 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:24.398969    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qxlf9" event={"ID":"b69b200d-32dc-4711-9127-b7a62b85a04e","Type":"ContainerStarted","Data":"2dc2c333e41458e28605c2c666d43a96585802b9f71566ece7a6a8288ed447c4"}
Jul 08 16:40:26.408518 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:26.408484    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qxlf9" event={"ID":"b69b200d-32dc-4711-9127-b7a62b85a04e","Type":"ContainerStarted","Data":"cf8972f5e1461b52ddcdc5f6b7c4c9b28a8d7bfcce22c38d9a5cc8fa693c82c0"}
Jul 08 16:40:26.408905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:26.408552    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-qxlf9"
Jul 08 16:40:26.423271 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:26.423222    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-qxlf9" podStartSLOduration=0.822847932 podStartE2EDuration="3.423207811s" totalImagesPullingTime="2.600359879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:40:26.423183489 +0000 UTC m=+341.959735971" watchObservedRunningTime="2026-07-08 16:40:26.423207811 +0000 UTC m=+341.959760295"
Jul 08 16:40:32.414850 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:40:32.414823    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-qxlf9"
Jul 08 16:41:49.335081 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:49.335047    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-x6dqm"]
Jul 08 16:41:49.338180 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:49.338163    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x6dqm"
Jul 08 16:41:49.348038 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:49.348018    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-x6dqm"]
Jul 08 16:41:49.446786 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:49.446766    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b7e784-74ba-491a-841a-2f252eb682c1-model-cache\") pod \"s3-init-x6dqm\" (UID: \"22b7e784-74ba-491a-841a-2f252eb682c1\") " pod="kserve/s3-init-x6dqm"
Jul 08 16:41:49.446884 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:49.446809    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t26xq\" (UniqueName: \"kubernetes.io/projected/22b7e784-74ba-491a-841a-2f252eb682c1-kube-api-access-t26xq\") pod \"s3-init-x6dqm\" (UID: \"22b7e784-74ba-491a-841a-2f252eb682c1\") " pod="kserve/s3-init-x6dqm"
Jul 08 16:41:49.547934 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:49.547913    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b7e784-74ba-491a-841a-2f252eb682c1-model-cache\") pod \"s3-init-x6dqm\" (UID: \"22b7e784-74ba-491a-841a-2f252eb682c1\") " pod="kserve/s3-init-x6dqm"
Jul 08 16:41:49.548027 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:49.547959    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t26xq\" (UniqueName: \"kubernetes.io/projected/22b7e784-74ba-491a-841a-2f252eb682c1-kube-api-access-t26xq\") pod \"s3-init-x6dqm\" (UID: \"22b7e784-74ba-491a-841a-2f252eb682c1\") " pod="kserve/s3-init-x6dqm"
Jul 08 16:41:49.548259 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:49.548241    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b7e784-74ba-491a-841a-2f252eb682c1-model-cache\") pod \"s3-init-x6dqm\" (UID: \"22b7e784-74ba-491a-841a-2f252eb682c1\") " pod="kserve/s3-init-x6dqm"
Jul 08 16:41:49.557081 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:49.557058    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t26xq\" (UniqueName: \"kubernetes.io/projected/22b7e784-74ba-491a-841a-2f252eb682c1-kube-api-access-t26xq\") pod \"s3-init-x6dqm\" (UID: \"22b7e784-74ba-491a-841a-2f252eb682c1\") " pod="kserve/s3-init-x6dqm"
Jul 08 16:41:49.646959 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:49.646911    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x6dqm"
Jul 08 16:41:49.771121 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:49.771014    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-x6dqm"]
Jul 08 16:41:49.773170 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:41:49.773144    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22b7e784_74ba_491a_841a_2f252eb682c1.slice/crio-8152a102498b7374824d35449a6dea9ae18e081191af44c4b37902f4cbb06830 WatchSource:0}: Error finding container 8152a102498b7374824d35449a6dea9ae18e081191af44c4b37902f4cbb06830: Status 404 returned error can't find the container with id 8152a102498b7374824d35449a6dea9ae18e081191af44c4b37902f4cbb06830
Jul 08 16:41:50.704701 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:50.704667    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6dqm" event={"ID":"22b7e784-74ba-491a-841a-2f252eb682c1","Type":"ContainerStarted","Data":"8152a102498b7374824d35449a6dea9ae18e081191af44c4b37902f4cbb06830"}
Jul 08 16:41:53.719921 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:53.719891    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6dqm" event={"ID":"22b7e784-74ba-491a-841a-2f252eb682c1","Type":"ContainerStarted","Data":"d82bf1428ca12258975498adc7b0c8478b00e08f3860119f2791fe39b15e6924"}
Jul 08 16:41:56.733133 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:56.733045    2550 generic.go:356] "Generic (PLEG): container finished" podID="22b7e784-74ba-491a-841a-2f252eb682c1" containerID="d82bf1428ca12258975498adc7b0c8478b00e08f3860119f2791fe39b15e6924" exitCode=0
Jul 08 16:41:56.733628 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:41:56.733119    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6dqm" event={"ID":"22b7e784-74ba-491a-841a-2f252eb682c1","Type":"ContainerDied","Data":"d82bf1428ca12258975498adc7b0c8478b00e08f3860119f2791fe39b15e6924"}
Jul 08 16:42:01.758148 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:42:01.758056    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6dqm" event={"ID":"22b7e784-74ba-491a-841a-2f252eb682c1","Type":"ContainerStarted","Data":"cfc40caacef52b9b4c8ab58c6c672fb9a14779af7dea10a84f1a11717de22e5d"}
Jul 08 16:42:01.779081 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:42:01.779029    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-x6dqm" podStartSLOduration=1.467964431 podStartE2EDuration="12.779009811s" totalImagesPullingTime="8.644934139s" totalInitContainerRuntime="2.666111241s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:41:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:42:01.777405302 +0000 UTC m=+437.313957787" watchObservedRunningTime="2026-07-08 16:42:01.779009811 +0000 UTC m=+437.315562297"
Jul 08 16:43:06.992844 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:06.992809    2550 generic.go:356] "Generic (PLEG): container finished" podID="22b7e784-74ba-491a-841a-2f252eb682c1" containerID="cfc40caacef52b9b4c8ab58c6c672fb9a14779af7dea10a84f1a11717de22e5d" exitCode=0
Jul 08 16:43:06.993538 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:06.992893    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6dqm" event={"ID":"22b7e784-74ba-491a-841a-2f252eb682c1","Type":"ContainerDied","Data":"cfc40caacef52b9b4c8ab58c6c672fb9a14779af7dea10a84f1a11717de22e5d"}
Jul 08 16:43:08.118904 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:08.118881    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x6dqm"
Jul 08 16:43:08.202881 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:08.202859    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t26xq\" (UniqueName: \"kubernetes.io/projected/22b7e784-74ba-491a-841a-2f252eb682c1-kube-api-access-t26xq\") pod \"22b7e784-74ba-491a-841a-2f252eb682c1\" (UID: \"22b7e784-74ba-491a-841a-2f252eb682c1\") "
Jul 08 16:43:08.203015 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:08.202921    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b7e784-74ba-491a-841a-2f252eb682c1-model-cache\") pod \"22b7e784-74ba-491a-841a-2f252eb682c1\" (UID: \"22b7e784-74ba-491a-841a-2f252eb682c1\") "
Jul 08 16:43:08.205014 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:08.204988    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b7e784-74ba-491a-841a-2f252eb682c1-kube-api-access-t26xq" (OuterVolumeSpecName: "kube-api-access-t26xq") pod "22b7e784-74ba-491a-841a-2f252eb682c1" (UID: "22b7e784-74ba-491a-841a-2f252eb682c1"). InnerVolumeSpecName "kube-api-access-t26xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:43:08.220764 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:08.220743    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b7e784-74ba-491a-841a-2f252eb682c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "22b7e784-74ba-491a-841a-2f252eb682c1" (UID: "22b7e784-74ba-491a-841a-2f252eb682c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:43:08.303745 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:08.303727    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t26xq\" (UniqueName: \"kubernetes.io/projected/22b7e784-74ba-491a-841a-2f252eb682c1-kube-api-access-t26xq\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:43:08.303817 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:08.303746    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b7e784-74ba-491a-841a-2f252eb682c1-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:43:09.001513 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:09.001496    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x6dqm"
Jul 08 16:43:09.001633 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:09.001527    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6dqm" event={"ID":"22b7e784-74ba-491a-841a-2f252eb682c1","Type":"ContainerDied","Data":"8152a102498b7374824d35449a6dea9ae18e081191af44c4b37902f4cbb06830"}
Jul 08 16:43:09.001633 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:09.001578    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8152a102498b7374824d35449a6dea9ae18e081191af44c4b37902f4cbb06830"
Jul 08 16:43:20.877467 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.877437    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"]
Jul 08 16:43:20.877840 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.877749    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b7e784-74ba-491a-841a-2f252eb682c1" containerName="download-hf-model"
Jul 08 16:43:20.877840 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.877762    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b7e784-74ba-491a-841a-2f252eb682c1" containerName="download-hf-model"
Jul 08 16:43:20.877840 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.877770    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b7e784-74ba-491a-841a-2f252eb682c1" containerName="s3-init"
Jul 08 16:43:20.877840 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.877775    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b7e784-74ba-491a-841a-2f252eb682c1" containerName="s3-init"
Jul 08 16:43:20.877840 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.877836    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b7e784-74ba-491a-841a-2f252eb682c1" containerName="s3-init"
Jul 08 16:43:20.883198 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.883167    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:20.885696 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.885674    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-f4d1e-predictor-serving-cert\""
Jul 08 16:43:20.885827 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.885680    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:43:20.885903 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.885824    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:43:20.885903 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.885739    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-f4d1e-kube-rbac-proxy-sar-config\""
Jul 08 16:43:20.886006 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.885766    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7vdnq\""
Jul 08 16:43:20.886608 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.886554    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:43:20.886684 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.886633    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:43:20.891166 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.891146    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"]
Jul 08 16:43:20.995626 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.995596    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20a82567-af40-4463-be50-bdbd939db669-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:20.995750 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.995634    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp98l\" (UniqueName: \"kubernetes.io/projected/20a82567-af40-4463-be50-bdbd939db669-kube-api-access-tp98l\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:20.995750 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.995666    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a82567-af40-4463-be50-bdbd939db669-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:20.995750 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.995726    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a82567-af40-4463-be50-bdbd939db669-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:20.995864 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:20.995787    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a82567-af40-4463-be50-bdbd939db669-proxy-tls\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:21.096065 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:21.096040    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp98l\" (UniqueName: \"kubernetes.io/projected/20a82567-af40-4463-be50-bdbd939db669-kube-api-access-tp98l\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:21.096163 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:21.096072    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a82567-af40-4463-be50-bdbd939db669-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:21.096163 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:21.096091    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a82567-af40-4463-be50-bdbd939db669-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:21.096163 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:21.096106    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a82567-af40-4463-be50-bdbd939db669-proxy-tls\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:21.096281 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:21.096171    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20a82567-af40-4463-be50-bdbd939db669-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:21.096508 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:21.096487    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a82567-af40-4463-be50-bdbd939db669-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:21.096954 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:21.096865    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a82567-af40-4463-be50-bdbd939db669-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:21.096954 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:21.096865    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20a82567-af40-4463-be50-bdbd939db669-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:21.098549 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:21.098531    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a82567-af40-4463-be50-bdbd939db669-proxy-tls\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:21.103809 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:21.103789    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp98l\" (UniqueName: \"kubernetes.io/projected/20a82567-af40-4463-be50-bdbd939db669-kube-api-access-tp98l\") pod \"isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:21.195354 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:21.195273    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:21.326799 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:21.326775    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"]
Jul 08 16:43:21.329176 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:43:21.329143    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a82567_af40_4463_be50_bdbd939db669.slice/crio-09ff8dc8f561ee0f3b6810eacaa0ae85eb88f25ea433e320151693040a6a6c87 WatchSource:0}: Error finding container 09ff8dc8f561ee0f3b6810eacaa0ae85eb88f25ea433e320151693040a6a6c87: Status 404 returned error can't find the container with id 09ff8dc8f561ee0f3b6810eacaa0ae85eb88f25ea433e320151693040a6a6c87
Jul 08 16:43:22.048639 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:22.048607    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" event={"ID":"20a82567-af40-4463-be50-bdbd939db669","Type":"ContainerStarted","Data":"32e0149146a85919efa7dea493d4a9a03b1b4947025d13fc966aa51fa3f44114"}
Jul 08 16:43:22.049018 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:22.048663    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" event={"ID":"20a82567-af40-4463-be50-bdbd939db669","Type":"ContainerStarted","Data":"09ff8dc8f561ee0f3b6810eacaa0ae85eb88f25ea433e320151693040a6a6c87"}
Jul 08 16:43:25.060430 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:25.060401    2550 generic.go:356] "Generic (PLEG): container finished" podID="20a82567-af40-4463-be50-bdbd939db669" containerID="32e0149146a85919efa7dea493d4a9a03b1b4947025d13fc966aa51fa3f44114" exitCode=0
Jul 08 16:43:25.060737 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:25.060471    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" event={"ID":"20a82567-af40-4463-be50-bdbd939db669","Type":"ContainerDied","Data":"32e0149146a85919efa7dea493d4a9a03b1b4947025d13fc966aa51fa3f44114"}
Jul 08 16:43:39.125056 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:39.125016    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" event={"ID":"20a82567-af40-4463-be50-bdbd939db669","Type":"ContainerStarted","Data":"0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458"}
Jul 08 16:43:41.135934 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:41.135903    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" event={"ID":"20a82567-af40-4463-be50-bdbd939db669","Type":"ContainerStarted","Data":"c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7"}
Jul 08 16:43:44.151479 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:44.151447    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" event={"ID":"20a82567-af40-4463-be50-bdbd939db669","Type":"ContainerStarted","Data":"c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d"}
Jul 08 16:43:44.151923 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:44.151684    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:44.151923 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:44.151752    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:44.151923 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:44.151788    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:44.152945 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:44.152906    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:43:44.153473 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:44.153452    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:43:44.174453 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:44.174415    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podStartSLOduration=1.992611477 podStartE2EDuration="24.174402119s" totalImagesPullingTime="18.551022973s" totalInitContainerRuntime="3.630767669s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:43:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:43:44.172189741 +0000 UTC m=+539.708742262" watchObservedRunningTime="2026-07-08 16:43:44.174402119 +0000 UTC m=+539.710954603"
Jul 08 16:43:45.155157 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:45.155120    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:43:45.155546 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:45.155434    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:43:45.158835 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:45.158817    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:43:46.159375 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:46.159339    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:43:46.159737 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:46.159602    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:43:56.159573 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:56.159533    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:43:56.159994 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:43:56.159972    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:44:06.159830 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:06.159787    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:44:06.160264 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:06.160230    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:44:16.159449 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:16.159408    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:44:16.159926 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:16.159803    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:44:26.160000 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:26.159960    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:44:26.160481 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:26.160458    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:44:36.159271 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:36.159230    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:44:36.159709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:36.159667    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:44:46.159685 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:46.159660    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:44:46.160030 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:46.159739    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:44:55.930846 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:55.930815    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"]
Jul 08 16:44:55.931338 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:55.931121    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" containerID="cri-o://0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458" gracePeriod=30
Jul 08 16:44:55.931338 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:55.931130    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" containerID="cri-o://c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d" gracePeriod=30
Jul 08 16:44:55.931338 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:55.931171    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kube-rbac-proxy" containerID="cri-o://c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7" gracePeriod=30
Jul 08 16:44:56.060317 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.060289    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"]
Jul 08 16:44:56.062851 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.062833    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.066189 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.066165    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-2fcf8-predictor-serving-cert\""
Jul 08 16:44:56.066450 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.066433    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-2fcf8-kube-rbac-proxy-sar-config\""
Jul 08 16:44:56.077019 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.076998    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"]
Jul 08 16:44:56.159535 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.159495    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:44:56.159855 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.159832    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:44:56.174376 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.174346    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"]
Jul 08 16:44:56.176865 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.176848    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.181184 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.181128    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-2fcf8-predictor-serving-cert\""
Jul 08 16:44:56.181184 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.181151    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-2fcf8-kube-rbac-proxy-sar-config\""
Jul 08 16:44:56.193516 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.193497    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"]
Jul 08 16:44:56.195254 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.195237    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-cabundle-cert\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.195351 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.195266    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.195351 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.195283    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.195440 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.195349    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-proxy-tls\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.195440 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.195425    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwm2g\" (UniqueName: \"kubernetes.io/projected/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kube-api-access-pwm2g\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.295833 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.295809    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-proxy-tls\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.295930 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.295840    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd077191-62eb-453a-b6a4-8dc7816b33e4-proxy-tls\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.295930 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.295860    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvh6l\" (UniqueName: \"kubernetes.io/projected/dd077191-62eb-453a-b6a4-8dc7816b33e4-kube-api-access-pvh6l\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.295930 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.295884    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwm2g\" (UniqueName: \"kubernetes.io/projected/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kube-api-access-pwm2g\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.295930 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.295910    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd077191-62eb-453a-b6a4-8dc7816b33e4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.296127 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.295955    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd077191-62eb-453a-b6a4-8dc7816b33e4-cabundle-cert\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.296127 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.295981    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-cabundle-cert\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.296127 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.295998    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd077191-62eb-453a-b6a4-8dc7816b33e4-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.296127 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.296021    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.296127 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.296045    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.296507 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.296482    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.296770 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.296698    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.296770 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.296721    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-cabundle-cert\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.298303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.298278    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-proxy-tls\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.304739 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.304714    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwm2g\" (UniqueName: \"kubernetes.io/projected/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kube-api-access-pwm2g\") pod \"isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.372676 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.372658    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:44:56.397306 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.397280    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd077191-62eb-453a-b6a4-8dc7816b33e4-proxy-tls\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.397411 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.397322    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvh6l\" (UniqueName: \"kubernetes.io/projected/dd077191-62eb-453a-b6a4-8dc7816b33e4-kube-api-access-pvh6l\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.397411 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.397367    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd077191-62eb-453a-b6a4-8dc7816b33e4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.397411 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.397395    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd077191-62eb-453a-b6a4-8dc7816b33e4-cabundle-cert\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.397581 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:44:56.397408    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-serving-cert: secret "isvc-xgboost-graph-raw-2fcf8-predictor-serving-cert" not found
Jul 08 16:44:56.397581 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.397437    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd077191-62eb-453a-b6a4-8dc7816b33e4-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.397581 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:44:56.397464    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd077191-62eb-453a-b6a4-8dc7816b33e4-proxy-tls podName:dd077191-62eb-453a-b6a4-8dc7816b33e4 nodeName:}" failed. No retries permitted until 2026-07-08 16:44:56.897448114 +0000 UTC m=+612.434000578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dd077191-62eb-453a-b6a4-8dc7816b33e4-proxy-tls") pod "isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" (UID: "dd077191-62eb-453a-b6a4-8dc7816b33e4") : secret "isvc-xgboost-graph-raw-2fcf8-predictor-serving-cert" not found
Jul 08 16:44:56.397818 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.397795    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd077191-62eb-453a-b6a4-8dc7816b33e4-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.398144 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.398122    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd077191-62eb-453a-b6a4-8dc7816b33e4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.398217 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.398201    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd077191-62eb-453a-b6a4-8dc7816b33e4-cabundle-cert\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.407481 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.407453    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvh6l\" (UniqueName: \"kubernetes.io/projected/dd077191-62eb-453a-b6a4-8dc7816b33e4-kube-api-access-pvh6l\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.421016 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.420951    2550 generic.go:356] "Generic (PLEG): container finished" podID="20a82567-af40-4463-be50-bdbd939db669" containerID="c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7" exitCode=2
Jul 08 16:44:56.421140 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.421010    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" event={"ID":"20a82567-af40-4463-be50-bdbd939db669","Type":"ContainerDied","Data":"c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7"}
Jul 08 16:44:56.500068 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.499934    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"]
Jul 08 16:44:56.502187 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:44:56.502160    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91dbb4fe_1419_4dc6_a3cd_1fa7807c1a40.slice/crio-69a2ab47d10e06f46f6be22088164b1e72502ddbd48f9b52e8bf48320ac428db WatchSource:0}: Error finding container 69a2ab47d10e06f46f6be22088164b1e72502ddbd48f9b52e8bf48320ac428db: Status 404 returned error can't find the container with id 69a2ab47d10e06f46f6be22088164b1e72502ddbd48f9b52e8bf48320ac428db
Jul 08 16:44:56.901856 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.901818    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd077191-62eb-453a-b6a4-8dc7816b33e4-proxy-tls\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:56.904144 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:56.904120    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd077191-62eb-453a-b6a4-8dc7816b33e4-proxy-tls\") pod \"isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:57.088373 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:57.088342    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:44:57.215846 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:57.215826    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"]
Jul 08 16:44:57.217881 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:44:57.217843    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd077191_62eb_453a_b6a4_8dc7816b33e4.slice/crio-a7aaffeb354412880191969588e0ea390fffe4686298a6532fb0ae8f9ea56e1e WatchSource:0}: Error finding container a7aaffeb354412880191969588e0ea390fffe4686298a6532fb0ae8f9ea56e1e: Status 404 returned error can't find the container with id a7aaffeb354412880191969588e0ea390fffe4686298a6532fb0ae8f9ea56e1e
Jul 08 16:44:57.428192 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:57.428089    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" event={"ID":"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40","Type":"ContainerStarted","Data":"1ed63094851a5287891fc49fbd036fb6fc8aef1c7b66b01e62dfe304f021ec51"}
Jul 08 16:44:57.428338 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:57.428244    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" event={"ID":"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40","Type":"ContainerStarted","Data":"69a2ab47d10e06f46f6be22088164b1e72502ddbd48f9b52e8bf48320ac428db"}
Jul 08 16:44:57.435625 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:57.435597    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" event={"ID":"dd077191-62eb-453a-b6a4-8dc7816b33e4","Type":"ContainerStarted","Data":"154b5d1923cae3eb2103a13a47206e495895d23655af919ce286d541c6f37db6"}
Jul 08 16:44:57.435766 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:44:57.435648    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" event={"ID":"dd077191-62eb-453a-b6a4-8dc7816b33e4","Type":"ContainerStarted","Data":"a7aaffeb354412880191969588e0ea390fffe4686298a6532fb0ae8f9ea56e1e"}
Jul 08 16:45:00.155640 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:00.155557    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 08 16:45:00.449429 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:00.449380    2550 generic.go:356] "Generic (PLEG): container finished" podID="20a82567-af40-4463-be50-bdbd939db669" containerID="0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458" exitCode=0
Jul 08 16:45:00.449511 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:00.449446    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" event={"ID":"20a82567-af40-4463-be50-bdbd939db669","Type":"ContainerDied","Data":"0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458"}
Jul 08 16:45:01.454517 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:01.454492    2550 generic.go:356] "Generic (PLEG): container finished" podID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerID="154b5d1923cae3eb2103a13a47206e495895d23655af919ce286d541c6f37db6" exitCode=0
Jul 08 16:45:01.454873 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:01.454587    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" event={"ID":"dd077191-62eb-453a-b6a4-8dc7816b33e4","Type":"ContainerDied","Data":"154b5d1923cae3eb2103a13a47206e495895d23655af919ce286d541c6f37db6"}
Jul 08 16:45:01.456330 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:01.456238    2550 generic.go:356] "Generic (PLEG): container finished" podID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerID="1ed63094851a5287891fc49fbd036fb6fc8aef1c7b66b01e62dfe304f021ec51" exitCode=0
Jul 08 16:45:01.456330 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:01.456280    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" event={"ID":"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40","Type":"ContainerDied","Data":"1ed63094851a5287891fc49fbd036fb6fc8aef1c7b66b01e62dfe304f021ec51"}
Jul 08 16:45:02.465396 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:02.465357    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" event={"ID":"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40","Type":"ContainerStarted","Data":"4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea"}
Jul 08 16:45:02.465841 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:02.465428    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" event={"ID":"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40","Type":"ContainerStarted","Data":"b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096"}
Jul 08 16:45:02.465901 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:02.465867    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:45:02.466097 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:02.466074    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:45:02.467268 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:02.467231    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:45:02.486997 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:02.486953    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podStartSLOduration=2.380949218 podStartE2EDuration="6.486932318s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.1059831s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:45:02.485467442 +0000 UTC m=+618.022019928" watchObservedRunningTime="2026-07-08 16:45:02.486932318 +0000 UTC m=+618.023484803"
Jul 08 16:45:03.470734 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:03.470671    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:45:05.155811 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:05.155773    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 08 16:45:06.159393 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:06.159307    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:45:06.159856 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:06.159673    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:45:08.476358 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:08.476331    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:45:08.476918 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:08.476887    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:45:10.155532 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:10.155495    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 08 16:45:10.155999 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:10.155619    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:45:15.155698 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:15.155660    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 08 16:45:16.159336 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:16.159286    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:45:16.159755 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:16.159410    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:45:16.159755 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:16.159636    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:45:16.159755 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:16.159742    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:45:18.476832 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:18.476795    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:45:20.156164 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:20.156123    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 08 16:45:25.155739 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:25.155685    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 08 16:45:26.159874 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:26.159833    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:45:26.159874 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:26.159854    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent" probeResult="failure" output="Get \"http://10.133.0.23:9081/\": dial tcp 10.133.0.23:9081: connect: connection refused"
Jul 08 16:45:28.056163 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.056133    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:45:28.153946 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.153925    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a82567-af40-4463-be50-bdbd939db669-kserve-provision-location\") pod \"20a82567-af40-4463-be50-bdbd939db669\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") "
Jul 08 16:45:28.154062 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.153984    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp98l\" (UniqueName: \"kubernetes.io/projected/20a82567-af40-4463-be50-bdbd939db669-kube-api-access-tp98l\") pod \"20a82567-af40-4463-be50-bdbd939db669\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") "
Jul 08 16:45:28.154062 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.154047    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20a82567-af40-4463-be50-bdbd939db669-cabundle-cert\") pod \"20a82567-af40-4463-be50-bdbd939db669\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") "
Jul 08 16:45:28.154169 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.154081    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a82567-af40-4463-be50-bdbd939db669-kube-rbac-proxy-sar-config\") pod \"20a82567-af40-4463-be50-bdbd939db669\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") "
Jul 08 16:45:28.154169 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.154134    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a82567-af40-4463-be50-bdbd939db669-proxy-tls\") pod \"20a82567-af40-4463-be50-bdbd939db669\" (UID: \"20a82567-af40-4463-be50-bdbd939db669\") "
Jul 08 16:45:28.154298 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.154216    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a82567-af40-4463-be50-bdbd939db669-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20a82567-af40-4463-be50-bdbd939db669" (UID: "20a82567-af40-4463-be50-bdbd939db669"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:45:28.154412 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.154345    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a82567-af40-4463-be50-bdbd939db669-kserve-provision-location\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:45:28.154472 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.154455    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a82567-af40-4463-be50-bdbd939db669-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "20a82567-af40-4463-be50-bdbd939db669" (UID: "20a82567-af40-4463-be50-bdbd939db669"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:45:28.154590 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.154550    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a82567-af40-4463-be50-bdbd939db669-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "20a82567-af40-4463-be50-bdbd939db669" (UID: "20a82567-af40-4463-be50-bdbd939db669"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:45:28.156878 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.156852    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a82567-af40-4463-be50-bdbd939db669-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "20a82567-af40-4463-be50-bdbd939db669" (UID: "20a82567-af40-4463-be50-bdbd939db669"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:45:28.156878 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.156858    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a82567-af40-4463-be50-bdbd939db669-kube-api-access-tp98l" (OuterVolumeSpecName: "kube-api-access-tp98l") pod "20a82567-af40-4463-be50-bdbd939db669" (UID: "20a82567-af40-4463-be50-bdbd939db669"). InnerVolumeSpecName "kube-api-access-tp98l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:45:28.255339 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.255317    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a82567-af40-4463-be50-bdbd939db669-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:45:28.255464 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.255342    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp98l\" (UniqueName: \"kubernetes.io/projected/20a82567-af40-4463-be50-bdbd939db669-kube-api-access-tp98l\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:45:28.255464 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.255355    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20a82567-af40-4463-be50-bdbd939db669-cabundle-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:45:28.255464 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.255367    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a82567-af40-4463-be50-bdbd939db669-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:45:28.477848 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.477773    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:45:28.577387 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.577359    2550 generic.go:356] "Generic (PLEG): container finished" podID="20a82567-af40-4463-be50-bdbd939db669" containerID="c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d" exitCode=0
Jul 08 16:45:28.577527 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.577425    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"
Jul 08 16:45:28.577527 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.577444    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" event={"ID":"20a82567-af40-4463-be50-bdbd939db669","Type":"ContainerDied","Data":"c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d"}
Jul 08 16:45:28.577527 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.577509    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7" event={"ID":"20a82567-af40-4463-be50-bdbd939db669","Type":"ContainerDied","Data":"09ff8dc8f561ee0f3b6810eacaa0ae85eb88f25ea433e320151693040a6a6c87"}
Jul 08 16:45:28.577670 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.577554    2550 scope.go:117] "RemoveContainer" containerID="c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d"
Jul 08 16:45:28.579810 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.579787    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" event={"ID":"dd077191-62eb-453a-b6a4-8dc7816b33e4","Type":"ContainerStarted","Data":"25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471"}
Jul 08 16:45:28.579900 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.579854    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" event={"ID":"dd077191-62eb-453a-b6a4-8dc7816b33e4","Type":"ContainerStarted","Data":"a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508"}
Jul 08 16:45:28.580154 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.580139    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:45:28.580213 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.580192    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:45:28.581599 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.581550    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:45:28.586805 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.586673    2550 scope.go:117] "RemoveContainer" containerID="c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7"
Jul 08 16:45:28.593982 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.593965    2550 scope.go:117] "RemoveContainer" containerID="0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458"
Jul 08 16:45:28.600773 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.600757    2550 scope.go:117] "RemoveContainer" containerID="32e0149146a85919efa7dea493d4a9a03b1b4947025d13fc966aa51fa3f44114"
Jul 08 16:45:28.605964 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.605929    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" podStartSLOduration=1.856894509 podStartE2EDuration="32.605917403s" totalImagesPullingTime="26.697816113s" totalInitContainerRuntime="4.051206781s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:45:28.604593739 +0000 UTC m=+644.141146224" watchObservedRunningTime="2026-07-08 16:45:28.605917403 +0000 UTC m=+644.142469887"
Jul 08 16:45:28.608052 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.608039    2550 scope.go:117] "RemoveContainer" containerID="c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d"
Jul 08 16:45:28.608344 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:45:28.608318    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d\": container with ID starting with c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d not found: ID does not exist" containerID="c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d"
Jul 08 16:45:28.608388 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.608349    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d"} err="failed to get container status \"c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d\": rpc error: code = NotFound desc = could not find container \"c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d\": container with ID starting with c925a12af4fdcbebedf368d6b4f879c3d375a9a01fdf66e6a9af86c2a449e71d not found: ID does not exist"
Jul 08 16:45:28.608388 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.608364    2550 scope.go:117] "RemoveContainer" containerID="c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7"
Jul 08 16:45:28.608590 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:45:28.608573    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7\": container with ID starting with c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7 not found: ID does not exist" containerID="c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7"
Jul 08 16:45:28.608662 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.608593    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7"} err="failed to get container status \"c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7\": rpc error: code = NotFound desc = could not find container \"c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7\": container with ID starting with c2aa5eab77fc0b138d1d2d263b79f684616d170fc23b57c2021c2f4a055c79f7 not found: ID does not exist"
Jul 08 16:45:28.608662 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.608603    2550 scope.go:117] "RemoveContainer" containerID="0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458"
Jul 08 16:45:28.608803 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:45:28.608788    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458\": container with ID starting with 0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458 not found: ID does not exist" containerID="0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458"
Jul 08 16:45:28.608837 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.608806    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458"} err="failed to get container status \"0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458\": rpc error: code = NotFound desc = could not find container \"0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458\": container with ID starting with 0a108a3b9af507b28702a58f81451a2b361b1b78405b0a7aaab2927984e31458 not found: ID does not exist"
Jul 08 16:45:28.608837 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.608821    2550 scope.go:117] "RemoveContainer" containerID="32e0149146a85919efa7dea493d4a9a03b1b4947025d13fc966aa51fa3f44114"
Jul 08 16:45:28.609038 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:45:28.609019    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e0149146a85919efa7dea493d4a9a03b1b4947025d13fc966aa51fa3f44114\": container with ID starting with 32e0149146a85919efa7dea493d4a9a03b1b4947025d13fc966aa51fa3f44114 not found: ID does not exist" containerID="32e0149146a85919efa7dea493d4a9a03b1b4947025d13fc966aa51fa3f44114"
Jul 08 16:45:28.609038 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.609037    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e0149146a85919efa7dea493d4a9a03b1b4947025d13fc966aa51fa3f44114"} err="failed to get container status \"32e0149146a85919efa7dea493d4a9a03b1b4947025d13fc966aa51fa3f44114\": rpc error: code = NotFound desc = could not find container \"32e0149146a85919efa7dea493d4a9a03b1b4947025d13fc966aa51fa3f44114\": container with ID starting with 32e0149146a85919efa7dea493d4a9a03b1b4947025d13fc966aa51fa3f44114 not found: ID does not exist"
Jul 08 16:45:28.617244 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.617221    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"]
Jul 08 16:45:28.621769 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:28.621745    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f4d1e-predictor-69bc59bcd9-dwtb7"]
Jul 08 16:45:29.028039 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:29.028010    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a82567-af40-4463-be50-bdbd939db669" path="/var/lib/kubelet/pods/20a82567-af40-4463-be50-bdbd939db669/volumes"
Jul 08 16:45:29.584553 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:29.584521    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:45:34.588633 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:34.588604    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:45:34.589056 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:34.589031    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:45:38.477220 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:38.477178    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:45:44.589181 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:44.589141    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:45:48.477238 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:48.477203    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:45:54.589504 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:54.589469    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:45:58.477627 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:45:58.477555    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:46:04.589404 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:04.589366    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:46:08.477231 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:08.477195    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:46:14.589163 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:14.589122    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:46:18.478177 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:18.478149    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:46:24.589705 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:24.589678    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:46:46.307456 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.307385    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"]
Jul 08 16:46:46.307944 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.307698    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container" containerID="cri-o://b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096" gracePeriod=30
Jul 08 16:46:46.307944 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.307717    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kube-rbac-proxy" containerID="cri-o://4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea" gracePeriod=30
Jul 08 16:46:46.384829 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.384801    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"]
Jul 08 16:46:46.385239 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.385223    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container"
Jul 08 16:46:46.385315 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.385241    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container"
Jul 08 16:46:46.385315 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.385260    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent"
Jul 08 16:46:46.385315 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.385268    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent"
Jul 08 16:46:46.385315 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.385288    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="storage-initializer"
Jul 08 16:46:46.385315 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.385297    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="storage-initializer"
Jul 08 16:46:46.385315 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.385308    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kube-rbac-proxy"
Jul 08 16:46:46.385315 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.385316    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kube-rbac-proxy"
Jul 08 16:46:46.385695 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.385392    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kube-rbac-proxy"
Jul 08 16:46:46.385695 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.385402    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="kserve-container"
Jul 08 16:46:46.385695 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.385410    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a82567-af40-4463-be50-bdbd939db669" containerName="agent"
Jul 08 16:46:46.388362 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.388342    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.391004 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.390980    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-36ff6-predictor-serving-cert\""
Jul 08 16:46:46.391111 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.391025    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-36ff6-kube-rbac-proxy-sar-config\""
Jul 08 16:46:46.398166 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.398142    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"]
Jul 08 16:46:46.453967 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.453938    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"]
Jul 08 16:46:46.456850 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.456833    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.459245 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.459226    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-36ff6-predictor-serving-cert\""
Jul 08 16:46:46.459245 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.459242    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-36ff6-kube-rbac-proxy-sar-config\""
Jul 08 16:46:46.474806 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.474787    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"]
Jul 08 16:46:46.503641 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.503612    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"]
Jul 08 16:46:46.503882 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.503857    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kserve-container" containerID="cri-o://a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508" gracePeriod=30
Jul 08 16:46:46.503962 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.503905    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kube-rbac-proxy" containerID="cri-o://25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471" gracePeriod=30
Jul 08 16:46:46.507654 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.507632    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5134f5e-e707-4030-806f-27609269d3fd-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.507761 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.507672    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5134f5e-e707-4030-806f-27609269d3fd-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.507761 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.507712    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz4w2\" (UniqueName: \"kubernetes.io/projected/b5134f5e-e707-4030-806f-27609269d3fd-kube-api-access-wz4w2\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.507761 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.507729    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5134f5e-e707-4030-806f-27609269d3fd-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.507878 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.507763    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5134f5e-e707-4030-806f-27609269d3fd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.608320 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.608270    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5134f5e-e707-4030-806f-27609269d3fd-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.608320 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.608305    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45276b19-ccea-435c-85be-ca6f51d77bd6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.608474 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.608340    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45276b19-ccea-435c-85be-ca6f51d77bd6-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.608474 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.608372    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5134f5e-e707-4030-806f-27609269d3fd-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.608474 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.608451    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zlcc\" (UniqueName: \"kubernetes.io/projected/45276b19-ccea-435c-85be-ca6f51d77bd6-kube-api-access-5zlcc\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.608665 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.608498    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45276b19-ccea-435c-85be-ca6f51d77bd6-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.608665 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.608594    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz4w2\" (UniqueName: \"kubernetes.io/projected/b5134f5e-e707-4030-806f-27609269d3fd-kube-api-access-wz4w2\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.608815 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.608662    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5134f5e-e707-4030-806f-27609269d3fd-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.608815 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.608692    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5134f5e-e707-4030-806f-27609269d3fd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.608815 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.608754    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45276b19-ccea-435c-85be-ca6f51d77bd6-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.609077 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.609044    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5134f5e-e707-4030-806f-27609269d3fd-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.609283 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.609259    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5134f5e-e707-4030-806f-27609269d3fd-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.609351 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.609333    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5134f5e-e707-4030-806f-27609269d3fd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.610928 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.610909    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5134f5e-e707-4030-806f-27609269d3fd-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.629950 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.629921    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz4w2\" (UniqueName: \"kubernetes.io/projected/b5134f5e-e707-4030-806f-27609269d3fd-kube-api-access-wz4w2\") pod \"isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.700903 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.700881    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:46.710012 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.709990    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45276b19-ccea-435c-85be-ca6f51d77bd6-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.710090 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.710064    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45276b19-ccea-435c-85be-ca6f51d77bd6-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.710149 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:46:46.710123    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-36ff6-predictor-serving-cert" not found
Jul 08 16:46:46.710200 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:46:46.710173    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45276b19-ccea-435c-85be-ca6f51d77bd6-proxy-tls podName:45276b19-ccea-435c-85be-ca6f51d77bd6 nodeName:}" failed. No retries permitted until 2026-07-08 16:46:47.210159226 +0000 UTC m=+722.746711691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45276b19-ccea-435c-85be-ca6f51d77bd6-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" (UID: "45276b19-ccea-435c-85be-ca6f51d77bd6") : secret "isvc-xgboost-graph-raw-hpa-36ff6-predictor-serving-cert" not found
Jul 08 16:46:46.710311 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.710103    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45276b19-ccea-435c-85be-ca6f51d77bd6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.710430 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.710344    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45276b19-ccea-435c-85be-ca6f51d77bd6-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.710430 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.710387    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zlcc\" (UniqueName: \"kubernetes.io/projected/45276b19-ccea-435c-85be-ca6f51d77bd6-kube-api-access-5zlcc\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.710536 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.710498    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45276b19-ccea-435c-85be-ca6f51d77bd6-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.710863 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.710843    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45276b19-ccea-435c-85be-ca6f51d77bd6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.710939 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.710914    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45276b19-ccea-435c-85be-ca6f51d77bd6-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.719118 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.719099    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zlcc\" (UniqueName: \"kubernetes.io/projected/45276b19-ccea-435c-85be-ca6f51d77bd6-kube-api-access-5zlcc\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:46.824086 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.824062    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"]
Jul 08 16:46:46.826274 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:46:46.826239    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5134f5e_e707_4030_806f_27609269d3fd.slice/crio-bfffa54bd82918763c11a1ab7d31b1d10843aca4c1391c14eb2adf406e7fb8af WatchSource:0}: Error finding container bfffa54bd82918763c11a1ab7d31b1d10843aca4c1391c14eb2adf406e7fb8af: Status 404 returned error can't find the container with id bfffa54bd82918763c11a1ab7d31b1d10843aca4c1391c14eb2adf406e7fb8af
Jul 08 16:46:46.878099 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.878067    2550 generic.go:356] "Generic (PLEG): container finished" podID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerID="25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471" exitCode=2
Jul 08 16:46:46.878183 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.878131    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" event={"ID":"dd077191-62eb-453a-b6a4-8dc7816b33e4","Type":"ContainerDied","Data":"25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471"}
Jul 08 16:46:46.879641 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.879613    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" event={"ID":"b5134f5e-e707-4030-806f-27609269d3fd","Type":"ContainerStarted","Data":"bfffa54bd82918763c11a1ab7d31b1d10843aca4c1391c14eb2adf406e7fb8af"}
Jul 08 16:46:46.881602 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.881585    2550 generic.go:356] "Generic (PLEG): container finished" podID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerID="4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea" exitCode=2
Jul 08 16:46:46.881677 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:46.881628    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" event={"ID":"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40","Type":"ContainerDied","Data":"4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea"}
Jul 08 16:46:47.214419 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:47.214349    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45276b19-ccea-435c-85be-ca6f51d77bd6-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:47.216864 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:47.216835    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45276b19-ccea-435c-85be-ca6f51d77bd6-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:47.369897 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:47.369858    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:47.513260 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:47.513185    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"]
Jul 08 16:46:47.519054 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:46:47.519026    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45276b19_ccea_435c_85be_ca6f51d77bd6.slice/crio-afb4fd0b9253a7ebaad575582b62271fbe996e1bcaaa5d5d476a9c7a5d92ce44 WatchSource:0}: Error finding container afb4fd0b9253a7ebaad575582b62271fbe996e1bcaaa5d5d476a9c7a5d92ce44: Status 404 returned error can't find the container with id afb4fd0b9253a7ebaad575582b62271fbe996e1bcaaa5d5d476a9c7a5d92ce44
Jul 08 16:46:47.887099 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:47.887051    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" event={"ID":"b5134f5e-e707-4030-806f-27609269d3fd","Type":"ContainerStarted","Data":"84ac428ccee51557b460ead8a390e723543a6ab511ce52d39e414229e2b34f38"}
Jul 08 16:46:47.888579 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:47.888544    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" event={"ID":"45276b19-ccea-435c-85be-ca6f51d77bd6","Type":"ContainerStarted","Data":"ea4e1128cbfedfdb4c29561c262f21b29ae7797043d0e412ee24be440e73d20a"}
Jul 08 16:46:47.888688 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:47.888599    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" event={"ID":"45276b19-ccea-435c-85be-ca6f51d77bd6","Type":"ContainerStarted","Data":"afb4fd0b9253a7ebaad575582b62271fbe996e1bcaaa5d5d476a9c7a5d92ce44"}
Jul 08 16:46:48.471725 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:48.471687    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 08 16:46:48.477112 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:48.477085    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:46:49.584937 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:49.584899    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 08 16:46:50.042057 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.042029    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:46:50.140642 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.140618    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd077191-62eb-453a-b6a4-8dc7816b33e4-cabundle-cert\") pod \"dd077191-62eb-453a-b6a4-8dc7816b33e4\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") "
Jul 08 16:46:50.140794 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.140653    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvh6l\" (UniqueName: \"kubernetes.io/projected/dd077191-62eb-453a-b6a4-8dc7816b33e4-kube-api-access-pvh6l\") pod \"dd077191-62eb-453a-b6a4-8dc7816b33e4\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") "
Jul 08 16:46:50.140794 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.140705    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd077191-62eb-453a-b6a4-8dc7816b33e4-kube-rbac-proxy-sar-config\") pod \"dd077191-62eb-453a-b6a4-8dc7816b33e4\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") "
Jul 08 16:46:50.140794 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.140732    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd077191-62eb-453a-b6a4-8dc7816b33e4-kserve-provision-location\") pod \"dd077191-62eb-453a-b6a4-8dc7816b33e4\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") "
Jul 08 16:46:50.140794 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.140794    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd077191-62eb-453a-b6a4-8dc7816b33e4-proxy-tls\") pod \"dd077191-62eb-453a-b6a4-8dc7816b33e4\" (UID: \"dd077191-62eb-453a-b6a4-8dc7816b33e4\") "
Jul 08 16:46:50.141071 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.141041    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd077191-62eb-453a-b6a4-8dc7816b33e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd077191-62eb-453a-b6a4-8dc7816b33e4" (UID: "dd077191-62eb-453a-b6a4-8dc7816b33e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:46:50.141166 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.141082    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd077191-62eb-453a-b6a4-8dc7816b33e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd077191-62eb-453a-b6a4-8dc7816b33e4" (UID: "dd077191-62eb-453a-b6a4-8dc7816b33e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:46:50.141166 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.141084    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd077191-62eb-453a-b6a4-8dc7816b33e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd077191-62eb-453a-b6a4-8dc7816b33e4" (UID: "dd077191-62eb-453a-b6a4-8dc7816b33e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:50.142871 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.142847    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd077191-62eb-453a-b6a4-8dc7816b33e4-kube-api-access-pvh6l" (OuterVolumeSpecName: "kube-api-access-pvh6l") pod "dd077191-62eb-453a-b6a4-8dc7816b33e4" (UID: "dd077191-62eb-453a-b6a4-8dc7816b33e4"). InnerVolumeSpecName "kube-api-access-pvh6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:46:50.143038 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.142900    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd077191-62eb-453a-b6a4-8dc7816b33e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd077191-62eb-453a-b6a4-8dc7816b33e4" (UID: "dd077191-62eb-453a-b6a4-8dc7816b33e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:46:50.241604 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.241545    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd077191-62eb-453a-b6a4-8dc7816b33e4-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:46:50.241604 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.241587    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd077191-62eb-453a-b6a4-8dc7816b33e4-cabundle-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:46:50.241604 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.241596    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvh6l\" (UniqueName: \"kubernetes.io/projected/dd077191-62eb-453a-b6a4-8dc7816b33e4-kube-api-access-pvh6l\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:46:50.241604 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.241605    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd077191-62eb-453a-b6a4-8dc7816b33e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:46:50.241792 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.241613    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd077191-62eb-453a-b6a4-8dc7816b33e4-kserve-provision-location\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:46:50.638286 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.638261    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:46:50.745878 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.745851    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwm2g\" (UniqueName: \"kubernetes.io/projected/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kube-api-access-pwm2g\") pod \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") "
Jul 08 16:46:50.746046 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.745892    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-cabundle-cert\") pod \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") "
Jul 08 16:46:50.746046 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.745913    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kube-rbac-proxy-sar-config\") pod \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") "
Jul 08 16:46:50.746046 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.745949    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kserve-provision-location\") pod \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") "
Jul 08 16:46:50.746046 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.746029    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-proxy-tls\") pod \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\" (UID: \"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40\") "
Jul 08 16:46:50.746258 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.746204    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" (UID: "91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:46:50.746308 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.746264    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-cabundle-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:46:50.746357 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.746304    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" (UID: "91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:50.746357 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.746330    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" (UID: "91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:46:50.748016 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.747997    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kube-api-access-pwm2g" (OuterVolumeSpecName: "kube-api-access-pwm2g") pod "91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" (UID: "91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40"). InnerVolumeSpecName "kube-api-access-pwm2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:46:50.748068 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.748042    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" (UID: "91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:46:50.847163 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.847136    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwm2g\" (UniqueName: \"kubernetes.io/projected/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kube-api-access-pwm2g\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:46:50.847163 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.847158    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:46:50.847281 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.847167    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-kserve-provision-location\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:46:50.847281 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.847177    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:46:50.902970 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.902944    2550 generic.go:356] "Generic (PLEG): container finished" podID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerID="b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096" exitCode=0
Jul 08 16:46:50.903067 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.903017    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"
Jul 08 16:46:50.903067 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.903036    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" event={"ID":"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40","Type":"ContainerDied","Data":"b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096"}
Jul 08 16:46:50.903164 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.903136    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc" event={"ID":"91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40","Type":"ContainerDied","Data":"69a2ab47d10e06f46f6be22088164b1e72502ddbd48f9b52e8bf48320ac428db"}
Jul 08 16:46:50.903215 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.903175    2550 scope.go:117] "RemoveContainer" containerID="4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea"
Jul 08 16:46:50.905142 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.905122    2550 generic.go:356] "Generic (PLEG): container finished" podID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerID="a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508" exitCode=0
Jul 08 16:46:50.905239 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.905205    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" event={"ID":"dd077191-62eb-453a-b6a4-8dc7816b33e4","Type":"ContainerDied","Data":"a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508"}
Jul 08 16:46:50.905294 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.905261    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc" event={"ID":"dd077191-62eb-453a-b6a4-8dc7816b33e4","Type":"ContainerDied","Data":"a7aaffeb354412880191969588e0ea390fffe4686298a6532fb0ae8f9ea56e1e"}
Jul 08 16:46:50.905294 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.905212    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"
Jul 08 16:46:50.906909 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.906883    2550 generic.go:356] "Generic (PLEG): container finished" podID="b5134f5e-e707-4030-806f-27609269d3fd" containerID="84ac428ccee51557b460ead8a390e723543a6ab511ce52d39e414229e2b34f38" exitCode=0
Jul 08 16:46:50.907012 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.906925    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" event={"ID":"b5134f5e-e707-4030-806f-27609269d3fd","Type":"ContainerDied","Data":"84ac428ccee51557b460ead8a390e723543a6ab511ce52d39e414229e2b34f38"}
Jul 08 16:46:50.911822 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.911805    2550 scope.go:117] "RemoveContainer" containerID="b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096"
Jul 08 16:46:50.922120 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.921948    2550 scope.go:117] "RemoveContainer" containerID="1ed63094851a5287891fc49fbd036fb6fc8aef1c7b66b01e62dfe304f021ec51"
Jul 08 16:46:50.932712 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.932672    2550 scope.go:117] "RemoveContainer" containerID="4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea"
Jul 08 16:46:50.933100 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:46:50.932973    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea\": container with ID starting with 4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea not found: ID does not exist" containerID="4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea"
Jul 08 16:46:50.933100 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.933011    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea"} err="failed to get container status \"4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea\": rpc error: code = NotFound desc = could not find container \"4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea\": container with ID starting with 4548edaf4635ba2da431b150beeafe184f2736d8a55f0a2def68b24796da48ea not found: ID does not exist"
Jul 08 16:46:50.933100 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.933033    2550 scope.go:117] "RemoveContainer" containerID="b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096"
Jul 08 16:46:50.933556 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:46:50.933497    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096\": container with ID starting with b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096 not found: ID does not exist" containerID="b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096"
Jul 08 16:46:50.933556 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.933532    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096"} err="failed to get container status \"b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096\": rpc error: code = NotFound desc = could not find container \"b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096\": container with ID starting with b6ec74374ef9c63deaacae6dd2d8c3429a2cdcf2ede53096b18e5e8d4c37d096 not found: ID does not exist"
Jul 08 16:46:50.933556 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.933548    2550 scope.go:117] "RemoveContainer" containerID="1ed63094851a5287891fc49fbd036fb6fc8aef1c7b66b01e62dfe304f021ec51"
Jul 08 16:46:50.933888 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:46:50.933871    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed63094851a5287891fc49fbd036fb6fc8aef1c7b66b01e62dfe304f021ec51\": container with ID starting with 1ed63094851a5287891fc49fbd036fb6fc8aef1c7b66b01e62dfe304f021ec51 not found: ID does not exist" containerID="1ed63094851a5287891fc49fbd036fb6fc8aef1c7b66b01e62dfe304f021ec51"
Jul 08 16:46:50.933943 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.933892    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed63094851a5287891fc49fbd036fb6fc8aef1c7b66b01e62dfe304f021ec51"} err="failed to get container status \"1ed63094851a5287891fc49fbd036fb6fc8aef1c7b66b01e62dfe304f021ec51\": rpc error: code = NotFound desc = could not find container \"1ed63094851a5287891fc49fbd036fb6fc8aef1c7b66b01e62dfe304f021ec51\": container with ID starting with 1ed63094851a5287891fc49fbd036fb6fc8aef1c7b66b01e62dfe304f021ec51 not found: ID does not exist"
Jul 08 16:46:50.933943 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.933910    2550 scope.go:117] "RemoveContainer" containerID="25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471"
Jul 08 16:46:50.946382 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.946361    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"]
Jul 08 16:46:50.950366 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.950347    2550 scope.go:117] "RemoveContainer" containerID="a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508"
Jul 08 16:46:50.950866 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.950848    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2fcf8-predictor-7b77855b97-nvnbc"]
Jul 08 16:46:50.961713 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.961692    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"]
Jul 08 16:46:50.965735 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.965715    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2fcf8-predictor-5b4c74cdfc-jwvpc"]
Jul 08 16:46:50.966005 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.965992    2550 scope.go:117] "RemoveContainer" containerID="154b5d1923cae3eb2103a13a47206e495895d23655af919ce286d541c6f37db6"
Jul 08 16:46:50.972953 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.972939    2550 scope.go:117] "RemoveContainer" containerID="25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471"
Jul 08 16:46:50.973231 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:46:50.973210    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471\": container with ID starting with 25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471 not found: ID does not exist" containerID="25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471"
Jul 08 16:46:50.973318 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.973235    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471"} err="failed to get container status \"25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471\": rpc error: code = NotFound desc = could not find container \"25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471\": container with ID starting with 25fc0aaa5a46c7f789eb9ad1a198c324e8528022a8f6b0f80f8cc67922f75471 not found: ID does not exist"
Jul 08 16:46:50.973318 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.973251    2550 scope.go:117] "RemoveContainer" containerID="a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508"
Jul 08 16:46:50.973507 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:46:50.973484    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508\": container with ID starting with a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508 not found: ID does not exist" containerID="a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508"
Jul 08 16:46:50.973545 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.973521    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508"} err="failed to get container status \"a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508\": rpc error: code = NotFound desc = could not find container \"a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508\": container with ID starting with a9b3a3ca32b26e4a86b4cd7bd760d3702f0285549bd246c91795369cddd68508 not found: ID does not exist"
Jul 08 16:46:50.973618 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.973544    2550 scope.go:117] "RemoveContainer" containerID="154b5d1923cae3eb2103a13a47206e495895d23655af919ce286d541c6f37db6"
Jul 08 16:46:50.973936 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:46:50.973920    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154b5d1923cae3eb2103a13a47206e495895d23655af919ce286d541c6f37db6\": container with ID starting with 154b5d1923cae3eb2103a13a47206e495895d23655af919ce286d541c6f37db6 not found: ID does not exist" containerID="154b5d1923cae3eb2103a13a47206e495895d23655af919ce286d541c6f37db6"
Jul 08 16:46:50.973985 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:50.973940    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154b5d1923cae3eb2103a13a47206e495895d23655af919ce286d541c6f37db6"} err="failed to get container status \"154b5d1923cae3eb2103a13a47206e495895d23655af919ce286d541c6f37db6\": rpc error: code = NotFound desc = could not find container \"154b5d1923cae3eb2103a13a47206e495895d23655af919ce286d541c6f37db6\": container with ID starting with 154b5d1923cae3eb2103a13a47206e495895d23655af919ce286d541c6f37db6 not found: ID does not exist"
Jul 08 16:46:51.030839 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:51.030814    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" path="/var/lib/kubelet/pods/91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40/volumes"
Jul 08 16:46:51.031516 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:51.031495    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" path="/var/lib/kubelet/pods/dd077191-62eb-453a-b6a4-8dc7816b33e4/volumes"
Jul 08 16:46:51.914538 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:51.914507    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" event={"ID":"b5134f5e-e707-4030-806f-27609269d3fd","Type":"ContainerStarted","Data":"df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366"}
Jul 08 16:46:51.914991 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:51.914590    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" event={"ID":"b5134f5e-e707-4030-806f-27609269d3fd","Type":"ContainerStarted","Data":"d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735"}
Jul 08 16:46:51.914991 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:51.914858    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:51.915091 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:51.915018    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:51.916269 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:51.916242    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:46:51.916929 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:51.916910    2550 generic.go:356] "Generic (PLEG): container finished" podID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerID="ea4e1128cbfedfdb4c29561c262f21b29ae7797043d0e412ee24be440e73d20a" exitCode=0
Jul 08 16:46:51.917017 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:51.916946    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" event={"ID":"45276b19-ccea-435c-85be-ca6f51d77bd6","Type":"ContainerDied","Data":"ea4e1128cbfedfdb4c29561c262f21b29ae7797043d0e412ee24be440e73d20a"}
Jul 08 16:46:51.935493 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:51.935455    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podStartSLOduration=2.365739235 podStartE2EDuration="5.935441748s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.569702513s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:51.934105258 +0000 UTC m=+727.470657756" watchObservedRunningTime="2026-07-08 16:46:51.935441748 +0000 UTC m=+727.471994233"
Jul 08 16:46:52.925306 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:52.925271    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" event={"ID":"45276b19-ccea-435c-85be-ca6f51d77bd6","Type":"ContainerStarted","Data":"33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277"}
Jul 08 16:46:52.925767 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:52.925332    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" event={"ID":"45276b19-ccea-435c-85be-ca6f51d77bd6","Type":"ContainerStarted","Data":"5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113"}
Jul 08 16:46:52.925767 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:52.925539    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:46:52.925917 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:52.925900    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:52.926080 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:52.926062    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:52.927394 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:52.927369    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:46:52.946305 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:52.946267    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" podStartSLOduration=3.490688934 podStartE2EDuration="6.94625507s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.455566136s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:52.944718803 +0000 UTC m=+728.481271289" watchObservedRunningTime="2026-07-08 16:46:52.94625507 +0000 UTC m=+728.482807554"
Jul 08 16:46:53.929938 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:53.929899    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:46:57.929936 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:57.929908    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:46:57.930498 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:57.930471    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:46:58.934290 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:58.934261    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:46:58.934807 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:46:58.934775    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:47:07.930949 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:47:07.930903    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:47:08.934998 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:47:08.934965    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:47:17.930515 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:47:17.930477    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:47:18.935088 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:47:18.935047    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:47:27.931330 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:47:27.931285    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:47:28.935193 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:47:28.935152    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:47:37.931448 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:47:37.931409    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:47:38.934999 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:47:38.934955    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:47:47.930377 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:47:47.930343    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:47:48.935586 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:47:48.935533    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:47:57.931540 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:47:57.931515    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:48:26.641849 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.641766    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"]
Jul 08 16:48:26.642318 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.642005    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container" containerID="cri-o://d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735" gracePeriod=30
Jul 08 16:48:26.642318 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.642111    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kube-rbac-proxy" containerID="cri-o://df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366" gracePeriod=30
Jul 08 16:48:26.662251 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:48:26.662215    2550 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5134f5e_e707_4030_806f_27609269d3fd.slice/crio-df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366.scope\": RecentStats: unable to find data in memory cache]"
Jul 08 16:48:26.678968 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.678940    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"]
Jul 08 16:48:26.679246 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679234    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container"
Jul 08 16:48:26.679303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679247    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container"
Jul 08 16:48:26.679303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679258    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="storage-initializer"
Jul 08 16:48:26.679303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679264    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="storage-initializer"
Jul 08 16:48:26.679303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679273    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kube-rbac-proxy"
Jul 08 16:48:26.679303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679278    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kube-rbac-proxy"
Jul 08 16:48:26.679303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679283    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kserve-container"
Jul 08 16:48:26.679303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679287    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kserve-container"
Jul 08 16:48:26.679303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679295    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kube-rbac-proxy"
Jul 08 16:48:26.679303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679301    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kube-rbac-proxy"
Jul 08 16:48:26.679788 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679311    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="storage-initializer"
Jul 08 16:48:26.679788 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679316    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="storage-initializer"
Jul 08 16:48:26.679788 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679365    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kserve-container"
Jul 08 16:48:26.679788 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679374    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kube-rbac-proxy"
Jul 08 16:48:26.679788 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679380    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="91dbb4fe-1419-4dc6-a3cd-1fa7807c1a40" containerName="kube-rbac-proxy"
Jul 08 16:48:26.679788 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.679386    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd077191-62eb-453a-b6a4-8dc7816b33e4" containerName="kserve-container"
Jul 08 16:48:26.682354 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.682340    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:26.684736 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.684716    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-4a3c0-predictor-serving-cert\""
Jul 08 16:48:26.684736 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.684727    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-4a3c0-kube-rbac-proxy-sar-config\""
Jul 08 16:48:26.691794 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.691775    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"]
Jul 08 16:48:26.738664 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.738641    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"]
Jul 08 16:48:26.738879 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.738857    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kserve-container" containerID="cri-o://5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113" gracePeriod=30
Jul 08 16:48:26.738966 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.738907    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kube-rbac-proxy" containerID="cri-o://33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277" gracePeriod=30
Jul 08 16:48:26.838870 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.838840    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0804c178-7641-4551-8a99-a57415fc4e5c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv\" (UID: \"0804c178-7641-4551-8a99-a57415fc4e5c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:26.838967 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.838886    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8msw\" (UniqueName: \"kubernetes.io/projected/0804c178-7641-4551-8a99-a57415fc4e5c-kube-api-access-m8msw\") pod \"message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv\" (UID: \"0804c178-7641-4551-8a99-a57415fc4e5c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:26.839010 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.838995    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0804c178-7641-4551-8a99-a57415fc4e5c-proxy-tls\") pod \"message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv\" (UID: \"0804c178-7641-4551-8a99-a57415fc4e5c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:26.940321 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.940240    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0804c178-7641-4551-8a99-a57415fc4e5c-proxy-tls\") pod \"message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv\" (UID: \"0804c178-7641-4551-8a99-a57415fc4e5c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:26.940321 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.940289    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0804c178-7641-4551-8a99-a57415fc4e5c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv\" (UID: \"0804c178-7641-4551-8a99-a57415fc4e5c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:26.940321 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.940310    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8msw\" (UniqueName: \"kubernetes.io/projected/0804c178-7641-4551-8a99-a57415fc4e5c-kube-api-access-m8msw\") pod \"message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv\" (UID: \"0804c178-7641-4551-8a99-a57415fc4e5c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:26.940961 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.940940    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0804c178-7641-4551-8a99-a57415fc4e5c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv\" (UID: \"0804c178-7641-4551-8a99-a57415fc4e5c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:26.942739 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.942712    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0804c178-7641-4551-8a99-a57415fc4e5c-proxy-tls\") pod \"message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv\" (UID: \"0804c178-7641-4551-8a99-a57415fc4e5c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:26.947617 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.947591    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8msw\" (UniqueName: \"kubernetes.io/projected/0804c178-7641-4551-8a99-a57415fc4e5c-kube-api-access-m8msw\") pod \"message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv\" (UID: \"0804c178-7641-4551-8a99-a57415fc4e5c\") " pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:26.994679 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:26.994644    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:27.119806 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:27.119672    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"]
Jul 08 16:48:27.122982 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:48:27.122957    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0804c178_7641_4551_8a99_a57415fc4e5c.slice/crio-a3dcb28bb47f2751b0e0ee2a55bb7b58c95aea9109b33900f802f1590ee8e764 WatchSource:0}: Error finding container a3dcb28bb47f2751b0e0ee2a55bb7b58c95aea9109b33900f802f1590ee8e764: Status 404 returned error can't find the container with id a3dcb28bb47f2751b0e0ee2a55bb7b58c95aea9109b33900f802f1590ee8e764
Jul 08 16:48:27.125224 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:27.125209    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:48:27.287229 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:27.287202    2550 generic.go:356] "Generic (PLEG): container finished" podID="b5134f5e-e707-4030-806f-27609269d3fd" containerID="df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366" exitCode=2
Jul 08 16:48:27.287344 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:27.287276    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" event={"ID":"b5134f5e-e707-4030-806f-27609269d3fd","Type":"ContainerDied","Data":"df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366"}
Jul 08 16:48:27.288384 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:27.288362    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv" event={"ID":"0804c178-7641-4551-8a99-a57415fc4e5c","Type":"ContainerStarted","Data":"a3dcb28bb47f2751b0e0ee2a55bb7b58c95aea9109b33900f802f1590ee8e764"}
Jul 08 16:48:27.290307 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:27.290289    2550 generic.go:356] "Generic (PLEG): container finished" podID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerID="33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277" exitCode=2
Jul 08 16:48:27.290386 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:27.290324    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" event={"ID":"45276b19-ccea-435c-85be-ca6f51d77bd6","Type":"ContainerDied","Data":"33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277"}
Jul 08 16:48:27.926399 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:27.926356    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 08 16:48:27.930734 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:27.930709    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:48:28.930267 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:28.930227    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 08 16:48:28.935024 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:28.934996    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:48:29.300835 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:29.300804    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv" event={"ID":"0804c178-7641-4551-8a99-a57415fc4e5c","Type":"ContainerStarted","Data":"4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26"}
Jul 08 16:48:29.301013 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:29.300860    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv" event={"ID":"0804c178-7641-4551-8a99-a57415fc4e5c","Type":"ContainerStarted","Data":"6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5"}
Jul 08 16:48:29.301076 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:29.301019    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:29.301166 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:29.301150    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:29.302825 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:29.302807    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:29.319338 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:29.319286    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv" podStartSLOduration=1.985269264 podStartE2EDuration="3.31927382s" totalImagesPullingTime="1.334004556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:48:29.317729841 +0000 UTC m=+824.854282327" watchObservedRunningTime="2026-07-08 16:48:29.31927382 +0000 UTC m=+824.855826361"
Jul 08 16:48:29.878194 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:29.878172    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:48:30.064457 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.064433    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zlcc\" (UniqueName: \"kubernetes.io/projected/45276b19-ccea-435c-85be-ca6f51d77bd6-kube-api-access-5zlcc\") pod \"45276b19-ccea-435c-85be-ca6f51d77bd6\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") "
Jul 08 16:48:30.064850 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.064463    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45276b19-ccea-435c-85be-ca6f51d77bd6-kube-rbac-proxy-sar-config\") pod \"45276b19-ccea-435c-85be-ca6f51d77bd6\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") "
Jul 08 16:48:30.064850 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.064505    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45276b19-ccea-435c-85be-ca6f51d77bd6-proxy-tls\") pod \"45276b19-ccea-435c-85be-ca6f51d77bd6\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") "
Jul 08 16:48:30.064850 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.064528    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45276b19-ccea-435c-85be-ca6f51d77bd6-cabundle-cert\") pod \"45276b19-ccea-435c-85be-ca6f51d77bd6\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") "
Jul 08 16:48:30.064850 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.064599    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45276b19-ccea-435c-85be-ca6f51d77bd6-kserve-provision-location\") pod \"45276b19-ccea-435c-85be-ca6f51d77bd6\" (UID: \"45276b19-ccea-435c-85be-ca6f51d77bd6\") "
Jul 08 16:48:30.065043 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.064853    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45276b19-ccea-435c-85be-ca6f51d77bd6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45276b19-ccea-435c-85be-ca6f51d77bd6" (UID: "45276b19-ccea-435c-85be-ca6f51d77bd6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:48:30.065043 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.064907    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45276b19-ccea-435c-85be-ca6f51d77bd6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45276b19-ccea-435c-85be-ca6f51d77bd6" (UID: "45276b19-ccea-435c-85be-ca6f51d77bd6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:48:30.065043 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.064961    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45276b19-ccea-435c-85be-ca6f51d77bd6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45276b19-ccea-435c-85be-ca6f51d77bd6" (UID: "45276b19-ccea-435c-85be-ca6f51d77bd6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:30.066663 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.066641    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45276b19-ccea-435c-85be-ca6f51d77bd6-kube-api-access-5zlcc" (OuterVolumeSpecName: "kube-api-access-5zlcc") pod "45276b19-ccea-435c-85be-ca6f51d77bd6" (UID: "45276b19-ccea-435c-85be-ca6f51d77bd6"). InnerVolumeSpecName "kube-api-access-5zlcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:48:30.066778 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.066759    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45276b19-ccea-435c-85be-ca6f51d77bd6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45276b19-ccea-435c-85be-ca6f51d77bd6" (UID: "45276b19-ccea-435c-85be-ca6f51d77bd6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:48:30.165715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.165691    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zlcc\" (UniqueName: \"kubernetes.io/projected/45276b19-ccea-435c-85be-ca6f51d77bd6-kube-api-access-5zlcc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:48:30.165715 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.165711    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45276b19-ccea-435c-85be-ca6f51d77bd6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:48:30.165845 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.165721    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45276b19-ccea-435c-85be-ca6f51d77bd6-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:48:30.165845 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.165730    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45276b19-ccea-435c-85be-ca6f51d77bd6-cabundle-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:48:30.165845 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.165737    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45276b19-ccea-435c-85be-ca6f51d77bd6-kserve-provision-location\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:48:30.306700 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.306669    2550 generic.go:356] "Generic (PLEG): container finished" podID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerID="5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113" exitCode=0
Jul 08 16:48:30.306790 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.306719    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"
Jul 08 16:48:30.306790 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.306746    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" event={"ID":"45276b19-ccea-435c-85be-ca6f51d77bd6","Type":"ContainerDied","Data":"5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113"}
Jul 08 16:48:30.306865 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.306808    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d" event={"ID":"45276b19-ccea-435c-85be-ca6f51d77bd6","Type":"ContainerDied","Data":"afb4fd0b9253a7ebaad575582b62271fbe996e1bcaaa5d5d476a9c7a5d92ce44"}
Jul 08 16:48:30.306865 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.306838    2550 scope.go:117] "RemoveContainer" containerID="33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277"
Jul 08 16:48:30.315003 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.314987    2550 scope.go:117] "RemoveContainer" containerID="5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113"
Jul 08 16:48:30.322194 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.322175    2550 scope.go:117] "RemoveContainer" containerID="ea4e1128cbfedfdb4c29561c262f21b29ae7797043d0e412ee24be440e73d20a"
Jul 08 16:48:30.329043 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.329029    2550 scope.go:117] "RemoveContainer" containerID="33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277"
Jul 08 16:48:30.329310 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:48:30.329292    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277\": container with ID starting with 33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277 not found: ID does not exist" containerID="33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277"
Jul 08 16:48:30.329352 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.329318    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277"} err="failed to get container status \"33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277\": rpc error: code = NotFound desc = could not find container \"33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277\": container with ID starting with 33f5c841adcb9f0d442cb70274f79a1d1bda379c4a80e5f19ece84828b7db277 not found: ID does not exist"
Jul 08 16:48:30.329352 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.329335    2550 scope.go:117] "RemoveContainer" containerID="5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113"
Jul 08 16:48:30.329602 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:48:30.329583    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113\": container with ID starting with 5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113 not found: ID does not exist" containerID="5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113"
Jul 08 16:48:30.329655 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.329607    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113"} err="failed to get container status \"5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113\": rpc error: code = NotFound desc = could not find container \"5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113\": container with ID starting with 5f3b4119dcf53830385bf353fbcd53eb58e451adb9fd26e2996797146e6f3113 not found: ID does not exist"
Jul 08 16:48:30.329655 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.329622    2550 scope.go:117] "RemoveContainer" containerID="ea4e1128cbfedfdb4c29561c262f21b29ae7797043d0e412ee24be440e73d20a"
Jul 08 16:48:30.329831 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:48:30.329818    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4e1128cbfedfdb4c29561c262f21b29ae7797043d0e412ee24be440e73d20a\": container with ID starting with ea4e1128cbfedfdb4c29561c262f21b29ae7797043d0e412ee24be440e73d20a not found: ID does not exist" containerID="ea4e1128cbfedfdb4c29561c262f21b29ae7797043d0e412ee24be440e73d20a"
Jul 08 16:48:30.329866 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.329835    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4e1128cbfedfdb4c29561c262f21b29ae7797043d0e412ee24be440e73d20a"} err="failed to get container status \"ea4e1128cbfedfdb4c29561c262f21b29ae7797043d0e412ee24be440e73d20a\": rpc error: code = NotFound desc = could not find container \"ea4e1128cbfedfdb4c29561c262f21b29ae7797043d0e412ee24be440e73d20a\": container with ID starting with ea4e1128cbfedfdb4c29561c262f21b29ae7797043d0e412ee24be440e73d20a not found: ID does not exist"
Jul 08 16:48:30.334176 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.334156    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"]
Jul 08 16:48:30.339150 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.339131    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-36ff6-predictor-5bdcd6b949-9wb9d"]
Jul 08 16:48:30.890050 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:30.890031    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:48:31.031651 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.031607    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" path="/var/lib/kubelet/pods/45276b19-ccea-435c-85be-ca6f51d77bd6/volumes"
Jul 08 16:48:31.074132 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.074099    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5134f5e-e707-4030-806f-27609269d3fd-kserve-provision-location\") pod \"b5134f5e-e707-4030-806f-27609269d3fd\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") "
Jul 08 16:48:31.074132 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.074138    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5134f5e-e707-4030-806f-27609269d3fd-proxy-tls\") pod \"b5134f5e-e707-4030-806f-27609269d3fd\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") "
Jul 08 16:48:31.074510 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.074172    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz4w2\" (UniqueName: \"kubernetes.io/projected/b5134f5e-e707-4030-806f-27609269d3fd-kube-api-access-wz4w2\") pod \"b5134f5e-e707-4030-806f-27609269d3fd\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") "
Jul 08 16:48:31.074510 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.074223    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5134f5e-e707-4030-806f-27609269d3fd-kube-rbac-proxy-sar-config\") pod \"b5134f5e-e707-4030-806f-27609269d3fd\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") "
Jul 08 16:48:31.074510 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.074245    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5134f5e-e707-4030-806f-27609269d3fd-cabundle-cert\") pod \"b5134f5e-e707-4030-806f-27609269d3fd\" (UID: \"b5134f5e-e707-4030-806f-27609269d3fd\") "
Jul 08 16:48:31.074510 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.074380    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5134f5e-e707-4030-806f-27609269d3fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5134f5e-e707-4030-806f-27609269d3fd" (UID: "b5134f5e-e707-4030-806f-27609269d3fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:31.074713 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.074611    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5134f5e-e707-4030-806f-27609269d3fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5134f5e-e707-4030-806f-27609269d3fd" (UID: "b5134f5e-e707-4030-806f-27609269d3fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:48:31.074713 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.074620    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5134f5e-e707-4030-806f-27609269d3fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5134f5e-e707-4030-806f-27609269d3fd" (UID: "b5134f5e-e707-4030-806f-27609269d3fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:48:31.076150 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.076129    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5134f5e-e707-4030-806f-27609269d3fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5134f5e-e707-4030-806f-27609269d3fd" (UID: "b5134f5e-e707-4030-806f-27609269d3fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:48:31.076237 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.076222    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5134f5e-e707-4030-806f-27609269d3fd-kube-api-access-wz4w2" (OuterVolumeSpecName: "kube-api-access-wz4w2") pod "b5134f5e-e707-4030-806f-27609269d3fd" (UID: "b5134f5e-e707-4030-806f-27609269d3fd"). InnerVolumeSpecName "kube-api-access-wz4w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:48:31.175684 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.175650    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5134f5e-e707-4030-806f-27609269d3fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:48:31.175684 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.175684    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5134f5e-e707-4030-806f-27609269d3fd-cabundle-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:48:31.175818 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.175700    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5134f5e-e707-4030-806f-27609269d3fd-kserve-provision-location\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:48:31.175818 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.175712    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5134f5e-e707-4030-806f-27609269d3fd-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:48:31.175818 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.175724    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz4w2\" (UniqueName: \"kubernetes.io/projected/b5134f5e-e707-4030-806f-27609269d3fd-kube-api-access-wz4w2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:48:31.312019 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.311966    2550 generic.go:356] "Generic (PLEG): container finished" podID="b5134f5e-e707-4030-806f-27609269d3fd" containerID="d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735" exitCode=0
Jul 08 16:48:31.312098 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.312022    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"
Jul 08 16:48:31.312098 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.312049    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" event={"ID":"b5134f5e-e707-4030-806f-27609269d3fd","Type":"ContainerDied","Data":"d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735"}
Jul 08 16:48:31.312192 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.312105    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z" event={"ID":"b5134f5e-e707-4030-806f-27609269d3fd","Type":"ContainerDied","Data":"bfffa54bd82918763c11a1ab7d31b1d10843aca4c1391c14eb2adf406e7fb8af"}
Jul 08 16:48:31.312192 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.312130    2550 scope.go:117] "RemoveContainer" containerID="df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366"
Jul 08 16:48:31.320455 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.320439    2550 scope.go:117] "RemoveContainer" containerID="d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735"
Jul 08 16:48:31.327543 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.327527    2550 scope.go:117] "RemoveContainer" containerID="84ac428ccee51557b460ead8a390e723543a6ab511ce52d39e414229e2b34f38"
Jul 08 16:48:31.333808 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.333790    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"]
Jul 08 16:48:31.335546 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.335484    2550 scope.go:117] "RemoveContainer" containerID="df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366"
Jul 08 16:48:31.335995 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:48:31.335881    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366\": container with ID starting with df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366 not found: ID does not exist" containerID="df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366"
Jul 08 16:48:31.335995 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.335913    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366"} err="failed to get container status \"df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366\": rpc error: code = NotFound desc = could not find container \"df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366\": container with ID starting with df93a06f1926ae6793a7978d9239bc954f06abfc1b37a0516f22173a754ca366 not found: ID does not exist"
Jul 08 16:48:31.335995 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.335933    2550 scope.go:117] "RemoveContainer" containerID="d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735"
Jul 08 16:48:31.336968 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:48:31.336948    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735\": container with ID starting with d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735 not found: ID does not exist" containerID="d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735"
Jul 08 16:48:31.337069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.336971    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735"} err="failed to get container status \"d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735\": rpc error: code = NotFound desc = could not find container \"d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735\": container with ID starting with d84035b791bcf0272cba48fac1e777f428edb25adac0185ddfa5e20b00f7f735 not found: ID does not exist"
Jul 08 16:48:31.337069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.336989    2550 scope.go:117] "RemoveContainer" containerID="84ac428ccee51557b460ead8a390e723543a6ab511ce52d39e414229e2b34f38"
Jul 08 16:48:31.337257 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:48:31.337230    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ac428ccee51557b460ead8a390e723543a6ab511ce52d39e414229e2b34f38\": container with ID starting with 84ac428ccee51557b460ead8a390e723543a6ab511ce52d39e414229e2b34f38 not found: ID does not exist" containerID="84ac428ccee51557b460ead8a390e723543a6ab511ce52d39e414229e2b34f38"
Jul 08 16:48:31.337298 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.337261    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ac428ccee51557b460ead8a390e723543a6ab511ce52d39e414229e2b34f38"} err="failed to get container status \"84ac428ccee51557b460ead8a390e723543a6ab511ce52d39e414229e2b34f38\": rpc error: code = NotFound desc = could not find container \"84ac428ccee51557b460ead8a390e723543a6ab511ce52d39e414229e2b34f38\": container with ID starting with 84ac428ccee51557b460ead8a390e723543a6ab511ce52d39e414229e2b34f38 not found: ID does not exist"
Jul 08 16:48:31.337785 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:31.337768    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-36ff6-predictor-5dc449fd6f-d6s4z"]
Jul 08 16:48:33.028549 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:33.028515    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5134f5e-e707-4030-806f-27609269d3fd" path="/var/lib/kubelet/pods/b5134f5e-e707-4030-806f-27609269d3fd/volumes"
Jul 08 16:48:36.317444 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.317417    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:48:36.698457 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698381    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"]
Jul 08 16:48:36.698685 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698673    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kube-rbac-proxy"
Jul 08 16:48:36.698685 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698686    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kube-rbac-proxy"
Jul 08 16:48:36.698759 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698698    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="storage-initializer"
Jul 08 16:48:36.698759 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698704    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="storage-initializer"
Jul 08 16:48:36.698759 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698719    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kube-rbac-proxy"
Jul 08 16:48:36.698759 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698724    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kube-rbac-proxy"
Jul 08 16:48:36.698759 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698731    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container"
Jul 08 16:48:36.698759 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698749    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container"
Jul 08 16:48:36.698759 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698755    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kserve-container"
Jul 08 16:48:36.698759 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698761    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kserve-container"
Jul 08 16:48:36.699007 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698767    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="storage-initializer"
Jul 08 16:48:36.699007 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698773    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="storage-initializer"
Jul 08 16:48:36.699007 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698820    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kube-rbac-proxy"
Jul 08 16:48:36.699007 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698828    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kserve-container"
Jul 08 16:48:36.699007 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698834    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="45276b19-ccea-435c-85be-ca6f51d77bd6" containerName="kube-rbac-proxy"
Jul 08 16:48:36.699007 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.698841    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5134f5e-e707-4030-806f-27609269d3fd" containerName="kserve-container"
Jul 08 16:48:36.703596 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.703581    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.705797 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.705775    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:48:36.705797 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.705789    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:48:36.705957 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.705778    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-4a3c0-kube-rbac-proxy-sar-config\""
Jul 08 16:48:36.706140 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.706118    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-4a3c0-predictor-serving-cert\""
Jul 08 16:48:36.711884 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.711860    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"]
Jul 08 16:48:36.712355 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.712331    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e43f258-b1d6-4e34-9461-5213d5a87e6d-cabundle-cert\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.712455 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.712427    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e43f258-b1d6-4e34-9461-5213d5a87e6d-proxy-tls\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.712512 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.712482    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9g57\" (UniqueName: \"kubernetes.io/projected/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kube-api-access-m9g57\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.712591 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.712514    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kserve-provision-location\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.712591 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.712573    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.813050 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.813026    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e43f258-b1d6-4e34-9461-5213d5a87e6d-proxy-tls\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.813196 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.813065    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9g57\" (UniqueName: \"kubernetes.io/projected/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kube-api-access-m9g57\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.813196 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.813085    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kserve-provision-location\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.813196 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.813106    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.813196 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.813128    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e43f258-b1d6-4e34-9461-5213d5a87e6d-cabundle-cert\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.813397 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:48:36.813198    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-serving-cert: secret "isvc-logger-raw-4a3c0-predictor-serving-cert" not found
Jul 08 16:48:36.813397 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:48:36.813371    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e43f258-b1d6-4e34-9461-5213d5a87e6d-proxy-tls podName:4e43f258-b1d6-4e34-9461-5213d5a87e6d nodeName:}" failed. No retries permitted until 2026-07-08 16:48:37.313349799 +0000 UTC m=+832.849902263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4e43f258-b1d6-4e34-9461-5213d5a87e6d-proxy-tls") pod "isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" (UID: "4e43f258-b1d6-4e34-9461-5213d5a87e6d") : secret "isvc-logger-raw-4a3c0-predictor-serving-cert" not found
Jul 08 16:48:36.813585 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.813546    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kserve-provision-location\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.813865 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.813849    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.813914 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.813865    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e43f258-b1d6-4e34-9461-5213d5a87e6d-cabundle-cert\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:36.821574 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:36.821547    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9g57\" (UniqueName: \"kubernetes.io/projected/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kube-api-access-m9g57\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:37.316717 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:37.316687    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e43f258-b1d6-4e34-9461-5213d5a87e6d-proxy-tls\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:37.318921 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:37.318904    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e43f258-b1d6-4e34-9461-5213d5a87e6d-proxy-tls\") pod \"isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:37.614487 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:37.614408    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:37.741160 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:37.741129    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"]
Jul 08 16:48:37.743265 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:48:37.743241    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e43f258_b1d6_4e34_9461_5213d5a87e6d.slice/crio-52a45360bc7e2561347d767d7a9c9a134f693285e4868dd8e8a20526b237e646 WatchSource:0}: Error finding container 52a45360bc7e2561347d767d7a9c9a134f693285e4868dd8e8a20526b237e646: Status 404 returned error can't find the container with id 52a45360bc7e2561347d767d7a9c9a134f693285e4868dd8e8a20526b237e646
Jul 08 16:48:38.344086 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:38.344049    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" event={"ID":"4e43f258-b1d6-4e34-9461-5213d5a87e6d","Type":"ContainerStarted","Data":"4d31c1a7c2189e97e2022782b26b6e54bce54259089c1928678594f63ead3ead"}
Jul 08 16:48:38.344433 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:38.344108    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" event={"ID":"4e43f258-b1d6-4e34-9461-5213d5a87e6d","Type":"ContainerStarted","Data":"52a45360bc7e2561347d767d7a9c9a134f693285e4868dd8e8a20526b237e646"}
Jul 08 16:48:42.359366 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:42.359328    2550 generic.go:356] "Generic (PLEG): container finished" podID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerID="4d31c1a7c2189e97e2022782b26b6e54bce54259089c1928678594f63ead3ead" exitCode=0
Jul 08 16:48:42.359712 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:42.359396    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" event={"ID":"4e43f258-b1d6-4e34-9461-5213d5a87e6d","Type":"ContainerDied","Data":"4d31c1a7c2189e97e2022782b26b6e54bce54259089c1928678594f63ead3ead"}
Jul 08 16:48:43.366040 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:43.366009    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" event={"ID":"4e43f258-b1d6-4e34-9461-5213d5a87e6d","Type":"ContainerStarted","Data":"51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b"}
Jul 08 16:48:43.366393 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:43.366070    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" event={"ID":"4e43f258-b1d6-4e34-9461-5213d5a87e6d","Type":"ContainerStarted","Data":"e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473"}
Jul 08 16:48:43.366393 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:43.366093    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" event={"ID":"4e43f258-b1d6-4e34-9461-5213d5a87e6d","Type":"ContainerStarted","Data":"a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe"}
Jul 08 16:48:43.366501 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:43.366436    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:43.366553 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:43.366502    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:43.367835 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:43.367801    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:48:43.387874 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:43.387831    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podStartSLOduration=3.544006459 podStartE2EDuration="7.387819376s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.843812917s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:48:43.386259884 +0000 UTC m=+838.922812369" watchObservedRunningTime="2026-07-08 16:48:43.387819376 +0000 UTC m=+838.924371860"
Jul 08 16:48:44.370742 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:44.370708    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:44.371097 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:44.370856    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:48:44.371950 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:44.371923    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:48:45.374794 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:45.374755    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:48:45.375155 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:45.375109    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:48:50.379104 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:50.379077    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:48:50.379590 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:50.379533    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:48:50.379924 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:48:50.379897    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:49:00.379626 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:49:00.379587    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:49:00.380189 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:49:00.380112    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:49:10.379996 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:49:10.379953    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:49:10.380441 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:49:10.380417    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:49:20.379933 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:49:20.379887    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:49:20.380457 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:49:20.380432    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:49:30.380039 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:49:30.380003    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:49:30.380547 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:49:30.380523    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:49:40.379826 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:49:40.379727    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:49:40.380287 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:49:40.380132    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:49:50.379744 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:49:50.379716    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:49:50.380187 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:49:50.380000    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:50:01.715837 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.715801    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv_0804c178-7641-4551-8a99-a57415fc4e5c/kserve-container/0.log"
Jul 08 16:50:01.855855 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.855821    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"]
Jul 08 16:50:01.856054 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.856028    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv" podUID="0804c178-7641-4551-8a99-a57415fc4e5c" containerName="kserve-container" containerID="cri-o://6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5" gracePeriod=30
Jul 08 16:50:01.856195 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.856089    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv" podUID="0804c178-7641-4551-8a99-a57415fc4e5c" containerName="kube-rbac-proxy" containerID="cri-o://4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26" gracePeriod=30
Jul 08 16:50:01.913790 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.913758    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"]
Jul 08 16:50:01.914087 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.914014    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" containerID="cri-o://a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe" gracePeriod=30
Jul 08 16:50:01.914158 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.914088    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent" containerID="cri-o://51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b" gracePeriod=30
Jul 08 16:50:01.914222 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.914084    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kube-rbac-proxy" containerID="cri-o://e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473" gracePeriod=30
Jul 08 16:50:01.982408 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.982350    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"]
Jul 08 16:50:01.985710 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.985692    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:01.988723 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.988704    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-f3f2c-predictor-serving-cert\""
Jul 08 16:50:01.989389 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.989371    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-f3f2c-kube-rbac-proxy-sar-config\""
Jul 08 16:50:01.998246 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:01.998227    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"]
Jul 08 16:50:02.117929 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.117894    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98kqg\" (UniqueName: \"kubernetes.io/projected/60f526cf-acaf-4d26-8487-a0b6489d531f-kube-api-access-98kqg\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.117929 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.117942    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60f526cf-acaf-4d26-8487-a0b6489d531f-cabundle-cert\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.118180 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.118162    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60f526cf-acaf-4d26-8487-a0b6489d531f-proxy-tls\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.118272 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.118200    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60f526cf-acaf-4d26-8487-a0b6489d531f-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.118272 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.118230    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60f526cf-acaf-4d26-8487-a0b6489d531f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.148646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.148626    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:50:02.219538 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.219506    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60f526cf-acaf-4d26-8487-a0b6489d531f-proxy-tls\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.219733 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.219552    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60f526cf-acaf-4d26-8487-a0b6489d531f-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.219733 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.219597    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60f526cf-acaf-4d26-8487-a0b6489d531f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.219733 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.219631    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98kqg\" (UniqueName: \"kubernetes.io/projected/60f526cf-acaf-4d26-8487-a0b6489d531f-kube-api-access-98kqg\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.219733 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.219660    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60f526cf-acaf-4d26-8487-a0b6489d531f-cabundle-cert\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.220020 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.219997    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60f526cf-acaf-4d26-8487-a0b6489d531f-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.220313 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.220290    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60f526cf-acaf-4d26-8487-a0b6489d531f-cabundle-cert\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.220436 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.220323    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60f526cf-acaf-4d26-8487-a0b6489d531f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.222113 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.222091    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60f526cf-acaf-4d26-8487-a0b6489d531f-proxy-tls\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.227623 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.227605    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98kqg\" (UniqueName: \"kubernetes.io/projected/60f526cf-acaf-4d26-8487-a0b6489d531f-kube-api-access-98kqg\") pod \"isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.295151 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.295125    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:02.320202 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.320174    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0804c178-7641-4551-8a99-a57415fc4e5c-kube-rbac-proxy-sar-config\") pod \"0804c178-7641-4551-8a99-a57415fc4e5c\" (UID: \"0804c178-7641-4551-8a99-a57415fc4e5c\") "
Jul 08 16:50:02.320310 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.320211    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8msw\" (UniqueName: \"kubernetes.io/projected/0804c178-7641-4551-8a99-a57415fc4e5c-kube-api-access-m8msw\") pod \"0804c178-7641-4551-8a99-a57415fc4e5c\" (UID: \"0804c178-7641-4551-8a99-a57415fc4e5c\") "
Jul 08 16:50:02.320310 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.320239    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0804c178-7641-4551-8a99-a57415fc4e5c-proxy-tls\") pod \"0804c178-7641-4551-8a99-a57415fc4e5c\" (UID: \"0804c178-7641-4551-8a99-a57415fc4e5c\") "
Jul 08 16:50:02.320578 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.320541    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0804c178-7641-4551-8a99-a57415fc4e5c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0804c178-7641-4551-8a99-a57415fc4e5c" (UID: "0804c178-7641-4551-8a99-a57415fc4e5c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:50:02.322312 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.322283    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0804c178-7641-4551-8a99-a57415fc4e5c-kube-api-access-m8msw" (OuterVolumeSpecName: "kube-api-access-m8msw") pod "0804c178-7641-4551-8a99-a57415fc4e5c" (UID: "0804c178-7641-4551-8a99-a57415fc4e5c"). InnerVolumeSpecName "kube-api-access-m8msw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:50:02.322403 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.322330    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0804c178-7641-4551-8a99-a57415fc4e5c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0804c178-7641-4551-8a99-a57415fc4e5c" (UID: "0804c178-7641-4551-8a99-a57415fc4e5c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:50:02.416378 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.416342    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"]
Jul 08 16:50:02.420407 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:50:02.420380    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f526cf_acaf_4d26_8487_a0b6489d531f.slice/crio-0ea2f9af0bf14bc8e7f3d6c0842845ff6750afb9793730b9958bccb05c7229d6 WatchSource:0}: Error finding container 0ea2f9af0bf14bc8e7f3d6c0842845ff6750afb9793730b9958bccb05c7229d6: Status 404 returned error can't find the container with id 0ea2f9af0bf14bc8e7f3d6c0842845ff6750afb9793730b9958bccb05c7229d6
Jul 08 16:50:02.421358 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.421335    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0804c178-7641-4551-8a99-a57415fc4e5c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:50:02.421445 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.421360    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8msw\" (UniqueName: \"kubernetes.io/projected/0804c178-7641-4551-8a99-a57415fc4e5c-kube-api-access-m8msw\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:50:02.421445 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.421375    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0804c178-7641-4551-8a99-a57415fc4e5c-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:50:02.670391 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.670309    2550 generic.go:356] "Generic (PLEG): container finished" podID="0804c178-7641-4551-8a99-a57415fc4e5c" containerID="4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26" exitCode=2
Jul 08 16:50:02.670391 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.670341    2550 generic.go:356] "Generic (PLEG): container finished" podID="0804c178-7641-4551-8a99-a57415fc4e5c" containerID="6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5" exitCode=2
Jul 08 16:50:02.670391 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.670349    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv" event={"ID":"0804c178-7641-4551-8a99-a57415fc4e5c","Type":"ContainerDied","Data":"4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26"}
Jul 08 16:50:02.670639 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.670413    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv" event={"ID":"0804c178-7641-4551-8a99-a57415fc4e5c","Type":"ContainerDied","Data":"6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5"}
Jul 08 16:50:02.670639 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.670436    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv" event={"ID":"0804c178-7641-4551-8a99-a57415fc4e5c","Type":"ContainerDied","Data":"a3dcb28bb47f2751b0e0ee2a55bb7b58c95aea9109b33900f802f1590ee8e764"}
Jul 08 16:50:02.670639 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.670361    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"
Jul 08 16:50:02.670639 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.670470    2550 scope.go:117] "RemoveContainer" containerID="4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26"
Jul 08 16:50:02.672327 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.672292    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" event={"ID":"60f526cf-acaf-4d26-8487-a0b6489d531f","Type":"ContainerStarted","Data":"cafded57f112aa0126db74aefac791d2572ae4af23b487d9d725c51bb8a01310"}
Jul 08 16:50:02.672445 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.672374    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" event={"ID":"60f526cf-acaf-4d26-8487-a0b6489d531f","Type":"ContainerStarted","Data":"0ea2f9af0bf14bc8e7f3d6c0842845ff6750afb9793730b9958bccb05c7229d6"}
Jul 08 16:50:02.675344 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.675319    2550 generic.go:356] "Generic (PLEG): container finished" podID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerID="e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473" exitCode=2
Jul 08 16:50:02.675452 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.675374    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" event={"ID":"4e43f258-b1d6-4e34-9461-5213d5a87e6d","Type":"ContainerDied","Data":"e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473"}
Jul 08 16:50:02.679664 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.679644    2550 scope.go:117] "RemoveContainer" containerID="6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5"
Jul 08 16:50:02.687750 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.687734    2550 scope.go:117] "RemoveContainer" containerID="4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26"
Jul 08 16:50:02.687987 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:50:02.687970    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26\": container with ID starting with 4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26 not found: ID does not exist" containerID="4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26"
Jul 08 16:50:02.688038 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.687995    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26"} err="failed to get container status \"4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26\": rpc error: code = NotFound desc = could not find container \"4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26\": container with ID starting with 4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26 not found: ID does not exist"
Jul 08 16:50:02.688038 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.688011    2550 scope.go:117] "RemoveContainer" containerID="6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5"
Jul 08 16:50:02.688250 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:50:02.688233    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5\": container with ID starting with 6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5 not found: ID does not exist" containerID="6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5"
Jul 08 16:50:02.688309 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.688258    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5"} err="failed to get container status \"6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5\": rpc error: code = NotFound desc = could not find container \"6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5\": container with ID starting with 6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5 not found: ID does not exist"
Jul 08 16:50:02.688309 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.688281    2550 scope.go:117] "RemoveContainer" containerID="4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26"
Jul 08 16:50:02.688526 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.688504    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26"} err="failed to get container status \"4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26\": rpc error: code = NotFound desc = could not find container \"4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26\": container with ID starting with 4bb1c886a25bc1c57269e327160008ebcb360c6b05472039ce354865f9e64b26 not found: ID does not exist"
Jul 08 16:50:02.688596 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.688527    2550 scope.go:117] "RemoveContainer" containerID="6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5"
Jul 08 16:50:02.688783 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.688765    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5"} err="failed to get container status \"6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5\": rpc error: code = NotFound desc = could not find container \"6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5\": container with ID starting with 6e4c0dc4f35268b6ed9acca1a17ae22aa1356dd2f5ab4429bceae5f20d5dada5 not found: ID does not exist"
Jul 08 16:50:02.706887 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.706861    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"]
Jul 08 16:50:02.709809 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:02.709787    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4a3c0-predictor-6c9d456b8c-c42dv"]
Jul 08 16:50:03.028086 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:03.028010    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0804c178-7641-4551-8a99-a57415fc4e5c" path="/var/lib/kubelet/pods/0804c178-7641-4551-8a99-a57415fc4e5c/volumes"
Jul 08 16:50:05.374976 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:05.374929    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 08 16:50:05.696091 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:05.696031    2550 generic.go:356] "Generic (PLEG): container finished" podID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerID="a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe" exitCode=0
Jul 08 16:50:05.696183 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:05.696105    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" event={"ID":"4e43f258-b1d6-4e34-9461-5213d5a87e6d","Type":"ContainerDied","Data":"a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe"}
Jul 08 16:50:06.701373 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:06.701341    2550 generic.go:356] "Generic (PLEG): container finished" podID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerID="cafded57f112aa0126db74aefac791d2572ae4af23b487d9d725c51bb8a01310" exitCode=0
Jul 08 16:50:06.701780 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:06.701414    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" event={"ID":"60f526cf-acaf-4d26-8487-a0b6489d531f","Type":"ContainerDied","Data":"cafded57f112aa0126db74aefac791d2572ae4af23b487d9d725c51bb8a01310"}
Jul 08 16:50:07.707444 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:07.707404    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" event={"ID":"60f526cf-acaf-4d26-8487-a0b6489d531f","Type":"ContainerStarted","Data":"70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25"}
Jul 08 16:50:07.707921 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:07.707481    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" event={"ID":"60f526cf-acaf-4d26-8487-a0b6489d531f","Type":"ContainerStarted","Data":"de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566"}
Jul 08 16:50:07.707921 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:07.707731    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:07.726549 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:07.726503    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podStartSLOduration=2.999649315 podStartE2EDuration="6.726488845s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.72683953s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:50:07.724729031 +0000 UTC m=+923.261281515" watchObservedRunningTime="2026-07-08 16:50:07.726488845 +0000 UTC m=+923.263041330"
Jul 08 16:50:08.712648 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:08.712615    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:08.713673 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:08.713647    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:50:09.716522 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:09.716483    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:50:10.375859 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:10.375815    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 08 16:50:10.380207 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:10.380179    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:50:10.380464 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:10.380441    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:50:14.720746 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:14.720719    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:50:14.721242 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:14.721153    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:50:15.374881 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:15.374844    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 08 16:50:15.375069 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:15.374951    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:50:20.375149 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:20.375107    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 08 16:50:20.380455 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:20.380426    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:50:20.380833 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:20.380813    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:50:24.721226 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:24.721184    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:50:25.375809 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:25.375772    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 08 16:50:30.374969 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:30.374929    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 08 16:50:30.380254 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:30.380229    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 16:50:30.380360 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:30.380338    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:50:30.380623 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:30.380599    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:50:30.380718 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:30.380701    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:50:32.093832 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.093812    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:50:32.120141 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.120116    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e43f258-b1d6-4e34-9461-5213d5a87e6d-cabundle-cert\") pod \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") "
Jul 08 16:50:32.120262 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.120185    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kserve-provision-location\") pod \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") "
Jul 08 16:50:32.120262 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.120237    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kube-rbac-proxy-sar-config\") pod \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") "
Jul 08 16:50:32.120369 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.120264    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e43f258-b1d6-4e34-9461-5213d5a87e6d-proxy-tls\") pod \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") "
Jul 08 16:50:32.120369 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.120301    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9g57\" (UniqueName: \"kubernetes.io/projected/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kube-api-access-m9g57\") pod \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\" (UID: \"4e43f258-b1d6-4e34-9461-5213d5a87e6d\") "
Jul 08 16:50:32.120537 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.120503    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e43f258-b1d6-4e34-9461-5213d5a87e6d" (UID: "4e43f258-b1d6-4e34-9461-5213d5a87e6d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:50:32.120676 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.120610    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e43f258-b1d6-4e34-9461-5213d5a87e6d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e43f258-b1d6-4e34-9461-5213d5a87e6d" (UID: "4e43f258-b1d6-4e34-9461-5213d5a87e6d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:50:32.120676 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.120622    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e43f258-b1d6-4e34-9461-5213d5a87e6d" (UID: "4e43f258-b1d6-4e34-9461-5213d5a87e6d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:50:32.122325 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.122302    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e43f258-b1d6-4e34-9461-5213d5a87e6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e43f258-b1d6-4e34-9461-5213d5a87e6d" (UID: "4e43f258-b1d6-4e34-9461-5213d5a87e6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:50:32.122428 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.122346    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kube-api-access-m9g57" (OuterVolumeSpecName: "kube-api-access-m9g57") pod "4e43f258-b1d6-4e34-9461-5213d5a87e6d" (UID: "4e43f258-b1d6-4e34-9461-5213d5a87e6d"). InnerVolumeSpecName "kube-api-access-m9g57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:50:32.220854 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.220803    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:50:32.220854 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.220825    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e43f258-b1d6-4e34-9461-5213d5a87e6d-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:50:32.220854 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.220835    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9g57\" (UniqueName: \"kubernetes.io/projected/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kube-api-access-m9g57\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:50:32.220854 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.220844    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e43f258-b1d6-4e34-9461-5213d5a87e6d-cabundle-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:50:32.220854 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.220852    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e43f258-b1d6-4e34-9461-5213d5a87e6d-kserve-provision-location\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:50:32.805738 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.805701    2550 generic.go:356] "Generic (PLEG): container finished" podID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerID="51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b" exitCode=0
Jul 08 16:50:32.805896 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.805767    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"
Jul 08 16:50:32.805896 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.805779    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" event={"ID":"4e43f258-b1d6-4e34-9461-5213d5a87e6d","Type":"ContainerDied","Data":"51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b"}
Jul 08 16:50:32.805896 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.805849    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58" event={"ID":"4e43f258-b1d6-4e34-9461-5213d5a87e6d","Type":"ContainerDied","Data":"52a45360bc7e2561347d767d7a9c9a134f693285e4868dd8e8a20526b237e646"}
Jul 08 16:50:32.805896 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.805879    2550 scope.go:117] "RemoveContainer" containerID="51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b"
Jul 08 16:50:32.814751 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.814733    2550 scope.go:117] "RemoveContainer" containerID="e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473"
Jul 08 16:50:32.822259 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.822239    2550 scope.go:117] "RemoveContainer" containerID="a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe"
Jul 08 16:50:32.827704 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.827681    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"]
Jul 08 16:50:32.830448 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.830430    2550 scope.go:117] "RemoveContainer" containerID="4d31c1a7c2189e97e2022782b26b6e54bce54259089c1928678594f63ead3ead"
Jul 08 16:50:32.831534 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.831517    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4a3c0-predictor-688f4c89f9-rwj58"]
Jul 08 16:50:32.837339 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.837320    2550 scope.go:117] "RemoveContainer" containerID="51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b"
Jul 08 16:50:32.837616 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:50:32.837597    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b\": container with ID starting with 51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b not found: ID does not exist" containerID="51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b"
Jul 08 16:50:32.837693 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.837628    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b"} err="failed to get container status \"51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b\": rpc error: code = NotFound desc = could not find container \"51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b\": container with ID starting with 51851d79da22c70e01e64a235a926ffc6ccda320a8c498735047df152976fa9b not found: ID does not exist"
Jul 08 16:50:32.837693 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.837653    2550 scope.go:117] "RemoveContainer" containerID="e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473"
Jul 08 16:50:32.837938 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:50:32.837920    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473\": container with ID starting with e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473 not found: ID does not exist" containerID="e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473"
Jul 08 16:50:32.837979 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.837945    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473"} err="failed to get container status \"e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473\": rpc error: code = NotFound desc = could not find container \"e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473\": container with ID starting with e4b924b21f1b974603d96756a477f75c3df6ab35a5da86341fc65652b70bf473 not found: ID does not exist"
Jul 08 16:50:32.837979 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.837962    2550 scope.go:117] "RemoveContainer" containerID="a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe"
Jul 08 16:50:32.838187 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:50:32.838171    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe\": container with ID starting with a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe not found: ID does not exist" containerID="a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe"
Jul 08 16:50:32.838227 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.838193    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe"} err="failed to get container status \"a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe\": rpc error: code = NotFound desc = could not find container \"a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe\": container with ID starting with a34835d5552c5dd2231763978c7cc928862b43335bbd65a0e2f6f4638b225cbe not found: ID does not exist"
Jul 08 16:50:32.838227 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.838208    2550 scope.go:117] "RemoveContainer" containerID="4d31c1a7c2189e97e2022782b26b6e54bce54259089c1928678594f63ead3ead"
Jul 08 16:50:32.838418 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:50:32.838401    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d31c1a7c2189e97e2022782b26b6e54bce54259089c1928678594f63ead3ead\": container with ID starting with 4d31c1a7c2189e97e2022782b26b6e54bce54259089c1928678594f63ead3ead not found: ID does not exist" containerID="4d31c1a7c2189e97e2022782b26b6e54bce54259089c1928678594f63ead3ead"
Jul 08 16:50:32.838457 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:32.838424    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d31c1a7c2189e97e2022782b26b6e54bce54259089c1928678594f63ead3ead"} err="failed to get container status \"4d31c1a7c2189e97e2022782b26b6e54bce54259089c1928678594f63ead3ead\": rpc error: code = NotFound desc = could not find container \"4d31c1a7c2189e97e2022782b26b6e54bce54259089c1928678594f63ead3ead\": container with ID starting with 4d31c1a7c2189e97e2022782b26b6e54bce54259089c1928678594f63ead3ead not found: ID does not exist"
Jul 08 16:50:33.032327 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:33.032299    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" path="/var/lib/kubelet/pods/4e43f258-b1d6-4e34-9461-5213d5a87e6d/volumes"
Jul 08 16:50:34.722149 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:34.722112    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:50:44.721625 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:44.721585    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:50:54.721677 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:50:54.721530    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:51:04.722175 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:51:04.722133    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:51:14.721823 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:51:14.721738    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:51:24.721626 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:51:24.721589    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:51:26.022976 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:51:26.022931    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:51:36.023830 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:51:36.023784    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:51:46.023003 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:51:46.022945    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:51:56.023904 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:51:56.023862    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:52:06.023790 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:06.023744    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:52:16.023495 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:16.023450    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:52:26.023713 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:26.023684    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:52:32.148841 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:32.148806    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"]
Jul 08 16:52:32.149291 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:32.149087    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" containerID="cri-o://de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566" gracePeriod=30
Jul 08 16:52:32.149291 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:32.149119    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kube-rbac-proxy" containerID="cri-o://70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25" gracePeriod=30
Jul 08 16:52:33.264832 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:33.264799    2550 generic.go:356] "Generic (PLEG): container finished" podID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerID="70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25" exitCode=2
Jul 08 16:52:33.265171 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:33.264867    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" event={"ID":"60f526cf-acaf-4d26-8487-a0b6489d531f","Type":"ContainerDied","Data":"70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25"}
Jul 08 16:52:34.717519 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:34.717481    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 08 16:52:36.023062 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:36.022969    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 16:52:37.326326 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326299    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"]
Jul 08 16:52:37.326711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326594    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="storage-initializer"
Jul 08 16:52:37.326711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326609    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="storage-initializer"
Jul 08 16:52:37.326711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326620    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kube-rbac-proxy"
Jul 08 16:52:37.326711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326625    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kube-rbac-proxy"
Jul 08 16:52:37.326711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326631    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent"
Jul 08 16:52:37.326711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326635    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent"
Jul 08 16:52:37.326711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326646    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0804c178-7641-4551-8a99-a57415fc4e5c" containerName="kserve-container"
Jul 08 16:52:37.326711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326651    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0804c178-7641-4551-8a99-a57415fc4e5c" containerName="kserve-container"
Jul 08 16:52:37.326711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326661    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container"
Jul 08 16:52:37.326711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326666    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container"
Jul 08 16:52:37.326711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326676    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0804c178-7641-4551-8a99-a57415fc4e5c" containerName="kube-rbac-proxy"
Jul 08 16:52:37.326711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326681    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0804c178-7641-4551-8a99-a57415fc4e5c" containerName="kube-rbac-proxy"
Jul 08 16:52:37.327104 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326730    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0804c178-7641-4551-8a99-a57415fc4e5c" containerName="kube-rbac-proxy"
Jul 08 16:52:37.327104 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326738    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kube-rbac-proxy"
Jul 08 16:52:37.327104 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326746    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="agent"
Jul 08 16:52:37.327104 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326751    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0804c178-7641-4551-8a99-a57415fc4e5c" containerName="kserve-container"
Jul 08 16:52:37.327104 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.326756    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e43f258-b1d6-4e34-9461-5213d5a87e6d" containerName="kserve-container"
Jul 08 16:52:37.329764 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.329749    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.332891 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.332871    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-b29bb6-predictor-serving-cert\""
Jul 08 16:52:37.333171 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.333153    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-b29bb6-kube-rbac-proxy-sar-config\""
Jul 08 16:52:37.349806 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.349784    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"]
Jul 08 16:52:37.384900 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.384874    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f82ad008-d4ee-471f-bf5d-7f5849645684-kserve-provision-location\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.384992 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.384909    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwx8x\" (UniqueName: \"kubernetes.io/projected/f82ad008-d4ee-471f-bf5d-7f5849645684-kube-api-access-cwx8x\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.384992 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.384935    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f82ad008-d4ee-471f-bf5d-7f5849645684-proxy-tls\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.385062 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.385004    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f82ad008-d4ee-471f-bf5d-7f5849645684-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.385062 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.385038    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f82ad008-d4ee-471f-bf5d-7f5849645684-cabundle-cert\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.485497 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.485463    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f82ad008-d4ee-471f-bf5d-7f5849645684-cabundle-cert\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.485664 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.485514    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f82ad008-d4ee-471f-bf5d-7f5849645684-kserve-provision-location\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.485664 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.485538    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwx8x\" (UniqueName: \"kubernetes.io/projected/f82ad008-d4ee-471f-bf5d-7f5849645684-kube-api-access-cwx8x\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.485664 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.485582    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f82ad008-d4ee-471f-bf5d-7f5849645684-proxy-tls\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.485664 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.485633    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f82ad008-d4ee-471f-bf5d-7f5849645684-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.485995 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.485966    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f82ad008-d4ee-471f-bf5d-7f5849645684-kserve-provision-location\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.486196 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.486177    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f82ad008-d4ee-471f-bf5d-7f5849645684-cabundle-cert\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.486255 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.486239    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f82ad008-d4ee-471f-bf5d-7f5849645684-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.488040 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.488021    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f82ad008-d4ee-471f-bf5d-7f5849645684-proxy-tls\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.498968 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.498949    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwx8x\" (UniqueName: \"kubernetes.io/projected/f82ad008-d4ee-471f-bf5d-7f5849645684-kube-api-access-cwx8x\") pod \"isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") " pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.639635 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.639529    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:37.767396 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:37.767372    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"]
Jul 08 16:52:37.768394 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:52:37.768371    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82ad008_d4ee_471f_bf5d_7f5849645684.slice/crio-9a744fb90d42bbbae16d12507ac91de76b7ecb0285e558f020ee7cea20e06b21 WatchSource:0}: Error finding container 9a744fb90d42bbbae16d12507ac91de76b7ecb0285e558f020ee7cea20e06b21: Status 404 returned error can't find the container with id 9a744fb90d42bbbae16d12507ac91de76b7ecb0285e558f020ee7cea20e06b21
Jul 08 16:52:38.283837 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:38.283807    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" event={"ID":"f82ad008-d4ee-471f-bf5d-7f5849645684","Type":"ContainerStarted","Data":"f8ca889206c7a7d6ea2e7ed9d962a337bdea4201879a72fa73e9a584405bf99b"}
Jul 08 16:52:38.284027 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:38.283859    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" event={"ID":"f82ad008-d4ee-471f-bf5d-7f5849645684","Type":"ContainerStarted","Data":"9a744fb90d42bbbae16d12507ac91de76b7ecb0285e558f020ee7cea20e06b21"}
Jul 08 16:52:39.717329 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:39.717285    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 08 16:52:40.786848 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.786828    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:52:40.808578 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.808538    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60f526cf-acaf-4d26-8487-a0b6489d531f-cabundle-cert\") pod \"60f526cf-acaf-4d26-8487-a0b6489d531f\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") "
Jul 08 16:52:40.808673 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.808608    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60f526cf-acaf-4d26-8487-a0b6489d531f-kserve-provision-location\") pod \"60f526cf-acaf-4d26-8487-a0b6489d531f\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") "
Jul 08 16:52:40.808673 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.808643    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60f526cf-acaf-4d26-8487-a0b6489d531f-proxy-tls\") pod \"60f526cf-acaf-4d26-8487-a0b6489d531f\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") "
Jul 08 16:52:40.808766 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.808696    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60f526cf-acaf-4d26-8487-a0b6489d531f-kube-rbac-proxy-sar-config\") pod \"60f526cf-acaf-4d26-8487-a0b6489d531f\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") "
Jul 08 16:52:40.808766 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.808761    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98kqg\" (UniqueName: \"kubernetes.io/projected/60f526cf-acaf-4d26-8487-a0b6489d531f-kube-api-access-98kqg\") pod \"60f526cf-acaf-4d26-8487-a0b6489d531f\" (UID: \"60f526cf-acaf-4d26-8487-a0b6489d531f\") "
Jul 08 16:52:40.808979 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.808952    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f526cf-acaf-4d26-8487-a0b6489d531f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "60f526cf-acaf-4d26-8487-a0b6489d531f" (UID: "60f526cf-acaf-4d26-8487-a0b6489d531f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:52:40.809110 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.808998    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f526cf-acaf-4d26-8487-a0b6489d531f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "60f526cf-acaf-4d26-8487-a0b6489d531f" (UID: "60f526cf-acaf-4d26-8487-a0b6489d531f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:52:40.809110 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.809088    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f526cf-acaf-4d26-8487-a0b6489d531f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "60f526cf-acaf-4d26-8487-a0b6489d531f" (UID: "60f526cf-acaf-4d26-8487-a0b6489d531f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:52:40.810876 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.810855    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f526cf-acaf-4d26-8487-a0b6489d531f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "60f526cf-acaf-4d26-8487-a0b6489d531f" (UID: "60f526cf-acaf-4d26-8487-a0b6489d531f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:52:40.811482 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.811446    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f526cf-acaf-4d26-8487-a0b6489d531f-kube-api-access-98kqg" (OuterVolumeSpecName: "kube-api-access-98kqg") pod "60f526cf-acaf-4d26-8487-a0b6489d531f" (UID: "60f526cf-acaf-4d26-8487-a0b6489d531f"). InnerVolumeSpecName "kube-api-access-98kqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:52:40.909786 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.909738    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60f526cf-acaf-4d26-8487-a0b6489d531f-kserve-provision-location\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:52:40.909786 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.909760    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60f526cf-acaf-4d26-8487-a0b6489d531f-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:52:40.909786 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.909771    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60f526cf-acaf-4d26-8487-a0b6489d531f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:52:40.909786 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.909780    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98kqg\" (UniqueName: \"kubernetes.io/projected/60f526cf-acaf-4d26-8487-a0b6489d531f-kube-api-access-98kqg\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:52:40.909786 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:40.909789    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60f526cf-acaf-4d26-8487-a0b6489d531f-cabundle-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:52:41.296750 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.296721    2550 generic.go:356] "Generic (PLEG): container finished" podID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerID="de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566" exitCode=0
Jul 08 16:52:41.296862 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.296783    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"
Jul 08 16:52:41.296862 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.296814    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" event={"ID":"60f526cf-acaf-4d26-8487-a0b6489d531f","Type":"ContainerDied","Data":"de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566"}
Jul 08 16:52:41.296960 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.296882    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56" event={"ID":"60f526cf-acaf-4d26-8487-a0b6489d531f","Type":"ContainerDied","Data":"0ea2f9af0bf14bc8e7f3d6c0842845ff6750afb9793730b9958bccb05c7229d6"}
Jul 08 16:52:41.296960 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.296919    2550 scope.go:117] "RemoveContainer" containerID="70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25"
Jul 08 16:52:41.298367 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.298347    2550 generic.go:356] "Generic (PLEG): container finished" podID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerID="f8ca889206c7a7d6ea2e7ed9d962a337bdea4201879a72fa73e9a584405bf99b" exitCode=0
Jul 08 16:52:41.298483 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.298425    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" event={"ID":"f82ad008-d4ee-471f-bf5d-7f5849645684","Type":"ContainerDied","Data":"f8ca889206c7a7d6ea2e7ed9d962a337bdea4201879a72fa73e9a584405bf99b"}
Jul 08 16:52:41.304862 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.304845    2550 scope.go:117] "RemoveContainer" containerID="de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566"
Jul 08 16:52:41.312292 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.312275    2550 scope.go:117] "RemoveContainer" containerID="cafded57f112aa0126db74aefac791d2572ae4af23b487d9d725c51bb8a01310"
Jul 08 16:52:41.321553 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.321476    2550 scope.go:117] "RemoveContainer" containerID="70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25"
Jul 08 16:52:41.322083 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:52:41.322042    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25\": container with ID starting with 70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25 not found: ID does not exist" containerID="70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25"
Jul 08 16:52:41.322180 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.322082    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25"} err="failed to get container status \"70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25\": rpc error: code = NotFound desc = could not find container \"70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25\": container with ID starting with 70626531791eae50e0f1d79b4c138abddc607f4af967e68414de3c1ec74d6f25 not found: ID does not exist"
Jul 08 16:52:41.322180 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.322104    2550 scope.go:117] "RemoveContainer" containerID="de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566"
Jul 08 16:52:41.322398 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:52:41.322379    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566\": container with ID starting with de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566 not found: ID does not exist" containerID="de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566"
Jul 08 16:52:41.322442 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.322406    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566"} err="failed to get container status \"de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566\": rpc error: code = NotFound desc = could not find container \"de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566\": container with ID starting with de3e8585d4d59224ca917724c6b93f8eef51111c288a7a81fdc80a917b594566 not found: ID does not exist"
Jul 08 16:52:41.322442 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.322425    2550 scope.go:117] "RemoveContainer" containerID="cafded57f112aa0126db74aefac791d2572ae4af23b487d9d725c51bb8a01310"
Jul 08 16:52:41.322689 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:52:41.322666    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cafded57f112aa0126db74aefac791d2572ae4af23b487d9d725c51bb8a01310\": container with ID starting with cafded57f112aa0126db74aefac791d2572ae4af23b487d9d725c51bb8a01310 not found: ID does not exist" containerID="cafded57f112aa0126db74aefac791d2572ae4af23b487d9d725c51bb8a01310"
Jul 08 16:52:41.322773 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.322696    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cafded57f112aa0126db74aefac791d2572ae4af23b487d9d725c51bb8a01310"} err="failed to get container status \"cafded57f112aa0126db74aefac791d2572ae4af23b487d9d725c51bb8a01310\": rpc error: code = NotFound desc = could not find container \"cafded57f112aa0126db74aefac791d2572ae4af23b487d9d725c51bb8a01310\": container with ID starting with cafded57f112aa0126db74aefac791d2572ae4af23b487d9d725c51bb8a01310 not found: ID does not exist"
Jul 08 16:52:41.333158 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.333138    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"]
Jul 08 16:52:41.337358 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:41.337334    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f3f2c-predictor-cf64f96dc-svx56"]
Jul 08 16:52:42.304148 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:42.304119    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" event={"ID":"f82ad008-d4ee-471f-bf5d-7f5849645684","Type":"ContainerStarted","Data":"85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65"}
Jul 08 16:52:42.304514 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:42.304186    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" event={"ID":"f82ad008-d4ee-471f-bf5d-7f5849645684","Type":"ContainerStarted","Data":"a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c"}
Jul 08 16:52:42.304514 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:42.304372    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:42.327426 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:42.327388    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" podStartSLOduration=1.976795424 podStartE2EDuration="5.327374763s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.350579339s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:52:42.326147318 +0000 UTC m=+1077.862699805" watchObservedRunningTime="2026-07-08 16:52:42.327374763 +0000 UTC m=+1077.863927247"
Jul 08 16:52:43.028525 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:43.028496    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" path="/var/lib/kubelet/pods/60f526cf-acaf-4d26-8487-a0b6489d531f/volumes"
Jul 08 16:52:43.308852 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:43.308827    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:43.309741 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:43.309717    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:52:44.313695 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:44.313650    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:52:49.318032 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:49.318008    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:52:49.318510 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:49.318485    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:52:59.318495 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:52:59.318451    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:53:09.319071 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:09.319030    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:53:19.318587 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:19.318526    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:53:29.318977 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:29.318933    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:53:39.318449 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:39.318412    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 16:53:49.318698 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:49.318666    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:53:57.586887 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.586853    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"]
Jul 08 16:53:57.587303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.587140    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kube-rbac-proxy"
Jul 08 16:53:57.587303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.587151    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kube-rbac-proxy"
Jul 08 16:53:57.587303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.587173    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container"
Jul 08 16:53:57.587303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.587177    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container"
Jul 08 16:53:57.587303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.587185    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="storage-initializer"
Jul 08 16:53:57.587303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.587190    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="storage-initializer"
Jul 08 16:53:57.587303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.587249    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kube-rbac-proxy"
Jul 08 16:53:57.587303 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.587258    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f526cf-acaf-4d26-8487-a0b6489d531f" containerName="kserve-container"
Jul 08 16:53:57.590280 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.590262    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.594687 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.594660    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-b29bb6\""
Jul 08 16:53:57.594796 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.594769    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-b29bb6-dockercfg-rxfrm\""
Jul 08 16:53:57.595208 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.595193    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b29bb6-kube-rbac-proxy-sar-config\""
Jul 08 16:53:57.595485 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.595463    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b29bb6-predictor-serving-cert\""
Jul 08 16:53:57.619642 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.619619    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"]
Jul 08 16:53:57.736584 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.736202    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kserve-provision-location\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.736584 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.736277    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-proxy-tls\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.736584 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.736310    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.736584 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.736366    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-cabundle-cert\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.736584 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.736392    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8f7k\" (UniqueName: \"kubernetes.io/projected/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kube-api-access-q8f7k\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.837785 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.837710    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kserve-provision-location\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.837785 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.837770    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-proxy-tls\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.838272 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.837798    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.838272 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.837840    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-cabundle-cert\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.838272 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.837963    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8f7k\" (UniqueName: \"kubernetes.io/projected/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kube-api-access-q8f7k\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.838272 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.838081    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kserve-provision-location\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.838516 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.838402    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.838516 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.838484    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-cabundle-cert\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.840279 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.840261    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-proxy-tls\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.851058 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.851032    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8f7k\" (UniqueName: \"kubernetes.io/projected/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kube-api-access-q8f7k\") pod \"isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") " pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:57.900088 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:57.900056    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:53:58.036275 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:58.036232    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"]
Jul 08 16:53:58.041148 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:53:58.041124    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecaee57b_2a3c_4351_bc3e_2e4e1c99652f.slice/crio-5dcce409ba525bd45d3f89c44ecfd6965e615da9b6cc00581a9294285f0e6154 WatchSource:0}: Error finding container 5dcce409ba525bd45d3f89c44ecfd6965e615da9b6cc00581a9294285f0e6154: Status 404 returned error can't find the container with id 5dcce409ba525bd45d3f89c44ecfd6965e615da9b6cc00581a9294285f0e6154
Jul 08 16:53:58.592375 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:58.592343    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms" event={"ID":"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f","Type":"ContainerStarted","Data":"1bf407afcc3a20ee20bf7dd79c48102fb04418d08a364c37e7b4e2eed45b4cbb"}
Jul 08 16:53:58.592745 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:53:58.592403    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms" event={"ID":"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f","Type":"ContainerStarted","Data":"5dcce409ba525bd45d3f89c44ecfd6965e615da9b6cc00581a9294285f0e6154"}
Jul 08 16:54:02.608620 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:02.608598    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms_ecaee57b-2a3c-4351-bc3e-2e4e1c99652f/storage-initializer/0.log"
Jul 08 16:54:02.608901 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:02.608629    2550 generic.go:356] "Generic (PLEG): container finished" podID="ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" containerID="1bf407afcc3a20ee20bf7dd79c48102fb04418d08a364c37e7b4e2eed45b4cbb" exitCode=1
Jul 08 16:54:02.608901 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:02.608702    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms" event={"ID":"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f","Type":"ContainerDied","Data":"1bf407afcc3a20ee20bf7dd79c48102fb04418d08a364c37e7b4e2eed45b4cbb"}
Jul 08 16:54:03.613642 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:03.613615    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms_ecaee57b-2a3c-4351-bc3e-2e4e1c99652f/storage-initializer/0.log"
Jul 08 16:54:03.614103 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:03.613679    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms" event={"ID":"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f","Type":"ContainerStarted","Data":"e777ad85a7247e0795f851282109043aac0a9d8c009126bf55736e6d55433cf1"}
Jul 08 16:54:08.632838 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:08.632817    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms_ecaee57b-2a3c-4351-bc3e-2e4e1c99652f/storage-initializer/1.log"
Jul 08 16:54:08.633188 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:08.633174    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms_ecaee57b-2a3c-4351-bc3e-2e4e1c99652f/storage-initializer/0.log"
Jul 08 16:54:08.633235 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:08.633205    2550 generic.go:356] "Generic (PLEG): container finished" podID="ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" containerID="e777ad85a7247e0795f851282109043aac0a9d8c009126bf55736e6d55433cf1" exitCode=1
Jul 08 16:54:08.633268 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:08.633258    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms" event={"ID":"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f","Type":"ContainerDied","Data":"e777ad85a7247e0795f851282109043aac0a9d8c009126bf55736e6d55433cf1"}
Jul 08 16:54:08.633318 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:08.633308    2550 scope.go:117] "RemoveContainer" containerID="1bf407afcc3a20ee20bf7dd79c48102fb04418d08a364c37e7b4e2eed45b4cbb"
Jul 08 16:54:08.633623 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:08.633598    2550 scope.go:117] "RemoveContainer" containerID="1bf407afcc3a20ee20bf7dd79c48102fb04418d08a364c37e7b4e2eed45b4cbb"
Jul 08 16:54:08.642169 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:54:08.642146    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms_kserve-ci-e2e-test(ecaee57b-2a3c-4351-bc3e-2e4e1c99652f)\"" pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms" podUID="ecaee57b-2a3c-4351-bc3e-2e4e1c99652f"
Jul 08 16:54:08.642313 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:54:08.642289    2550 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms_kserve-ci-e2e-test_ecaee57b-2a3c-4351-bc3e-2e4e1c99652f_0 in pod sandbox 5dcce409ba525bd45d3f89c44ecfd6965e615da9b6cc00581a9294285f0e6154 from index: no such id: '1bf407afcc3a20ee20bf7dd79c48102fb04418d08a364c37e7b4e2eed45b4cbb'" containerID="1bf407afcc3a20ee20bf7dd79c48102fb04418d08a364c37e7b4e2eed45b4cbb"
Jul 08 16:54:08.642401 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:08.642324    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf407afcc3a20ee20bf7dd79c48102fb04418d08a364c37e7b4e2eed45b4cbb"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms_kserve-ci-e2e-test_ecaee57b-2a3c-4351-bc3e-2e4e1c99652f_0 in pod sandbox 5dcce409ba525bd45d3f89c44ecfd6965e615da9b6cc00581a9294285f0e6154 from index: no such id: '1bf407afcc3a20ee20bf7dd79c48102fb04418d08a364c37e7b4e2eed45b4cbb'"
Jul 08 16:54:09.642658 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:09.642636    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms_ecaee57b-2a3c-4351-bc3e-2e4e1c99652f/storage-initializer/1.log"
Jul 08 16:54:13.583451 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.583421    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"]
Jul 08 16:54:13.649309 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.649279    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"]
Jul 08 16:54:13.649532 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.649506    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kserve-container" containerID="cri-o://a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c" gracePeriod=30
Jul 08 16:54:13.649612 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.649557    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kube-rbac-proxy" containerID="cri-o://85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65" gracePeriod=30
Jul 08 16:54:13.768488 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.768469    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms_ecaee57b-2a3c-4351-bc3e-2e4e1c99652f/storage-initializer/1.log"
Jul 08 16:54:13.769962 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.769947    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:54:13.851465 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.851395    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kube-rbac-proxy-sar-config\") pod \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") "
Jul 08 16:54:13.851465 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.851444    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-cabundle-cert\") pod \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") "
Jul 08 16:54:13.851646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.851496    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-proxy-tls\") pod \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") "
Jul 08 16:54:13.851646 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.851544    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8f7k\" (UniqueName: \"kubernetes.io/projected/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kube-api-access-q8f7k\") pod \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") "
Jul 08 16:54:13.851750 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.851647    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kserve-provision-location\") pod \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\" (UID: \"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f\") "
Jul 08 16:54:13.852200 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.852166    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" (UID: "ecaee57b-2a3c-4351-bc3e-2e4e1c99652f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:54:13.852487 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.852464    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" (UID: "ecaee57b-2a3c-4351-bc3e-2e4e1c99652f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:54:13.852813 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.852792    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" (UID: "ecaee57b-2a3c-4351-bc3e-2e4e1c99652f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:54:13.856717 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.856694    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" (UID: "ecaee57b-2a3c-4351-bc3e-2e4e1c99652f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:54:13.857077 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.856987    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kube-api-access-q8f7k" (OuterVolumeSpecName: "kube-api-access-q8f7k") pod "ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" (UID: "ecaee57b-2a3c-4351-bc3e-2e4e1c99652f"). InnerVolumeSpecName "kube-api-access-q8f7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:54:13.952669 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.952640    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kserve-provision-location\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:13.952669 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.952670    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:13.952669 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.952680    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-cabundle-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:13.952841 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.952689    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:13.952841 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:13.952700    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8f7k\" (UniqueName: \"kubernetes.io/projected/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f-kube-api-access-q8f7k\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:14.314209 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:14.314175    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 08 16:54:14.663248 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:14.663175    2550 generic.go:356] "Generic (PLEG): container finished" podID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerID="85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65" exitCode=2
Jul 08 16:54:14.663557 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:14.663250    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" event={"ID":"f82ad008-d4ee-471f-bf5d-7f5849645684","Type":"ContainerDied","Data":"85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65"}
Jul 08 16:54:14.664418 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:14.664401    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms_ecaee57b-2a3c-4351-bc3e-2e4e1c99652f/storage-initializer/1.log"
Jul 08 16:54:14.664479 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:14.664462    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms" event={"ID":"ecaee57b-2a3c-4351-bc3e-2e4e1c99652f","Type":"ContainerDied","Data":"5dcce409ba525bd45d3f89c44ecfd6965e615da9b6cc00581a9294285f0e6154"}
Jul 08 16:54:14.664528 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:14.664516    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"
Jul 08 16:54:14.664575 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:14.664530    2550 scope.go:117] "RemoveContainer" containerID="e777ad85a7247e0795f851282109043aac0a9d8c009126bf55736e6d55433cf1"
Jul 08 16:54:14.703150 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:14.703125    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"]
Jul 08 16:54:14.710456 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:14.710435    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b29bb6-predictor-7f7c99774c-f5lms"]
Jul 08 16:54:15.028489 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:15.028413    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" path="/var/lib/kubelet/pods/ecaee57b-2a3c-4351-bc3e-2e4e1c99652f/volumes"
Jul 08 16:54:17.487772 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.487744    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:54:17.578898 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.578821    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f82ad008-d4ee-471f-bf5d-7f5849645684-kserve-provision-location\") pod \"f82ad008-d4ee-471f-bf5d-7f5849645684\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") "
Jul 08 16:54:17.579061 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.578906    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f82ad008-d4ee-471f-bf5d-7f5849645684-kube-rbac-proxy-sar-config\") pod \"f82ad008-d4ee-471f-bf5d-7f5849645684\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") "
Jul 08 16:54:17.579061 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.578932    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f82ad008-d4ee-471f-bf5d-7f5849645684-proxy-tls\") pod \"f82ad008-d4ee-471f-bf5d-7f5849645684\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") "
Jul 08 16:54:17.579061 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.578957    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwx8x\" (UniqueName: \"kubernetes.io/projected/f82ad008-d4ee-471f-bf5d-7f5849645684-kube-api-access-cwx8x\") pod \"f82ad008-d4ee-471f-bf5d-7f5849645684\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") "
Jul 08 16:54:17.579061 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.579009    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f82ad008-d4ee-471f-bf5d-7f5849645684-cabundle-cert\") pod \"f82ad008-d4ee-471f-bf5d-7f5849645684\" (UID: \"f82ad008-d4ee-471f-bf5d-7f5849645684\") "
Jul 08 16:54:17.579268 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.579137    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82ad008-d4ee-471f-bf5d-7f5849645684-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f82ad008-d4ee-471f-bf5d-7f5849645684" (UID: "f82ad008-d4ee-471f-bf5d-7f5849645684"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:54:17.579268 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.579234    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82ad008-d4ee-471f-bf5d-7f5849645684-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f82ad008-d4ee-471f-bf5d-7f5849645684" (UID: "f82ad008-d4ee-471f-bf5d-7f5849645684"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:54:17.579395 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.579375    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82ad008-d4ee-471f-bf5d-7f5849645684-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f82ad008-d4ee-471f-bf5d-7f5849645684" (UID: "f82ad008-d4ee-471f-bf5d-7f5849645684"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:54:17.581057 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.581030    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f82ad008-d4ee-471f-bf5d-7f5849645684-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f82ad008-d4ee-471f-bf5d-7f5849645684" (UID: "f82ad008-d4ee-471f-bf5d-7f5849645684"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:54:17.581150 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.581127    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82ad008-d4ee-471f-bf5d-7f5849645684-kube-api-access-cwx8x" (OuterVolumeSpecName: "kube-api-access-cwx8x") pod "f82ad008-d4ee-471f-bf5d-7f5849645684" (UID: "f82ad008-d4ee-471f-bf5d-7f5849645684"). InnerVolumeSpecName "kube-api-access-cwx8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:54:17.678487 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.678460    2550 generic.go:356] "Generic (PLEG): container finished" podID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerID="a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c" exitCode=0
Jul 08 16:54:17.678610 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.678511    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" event={"ID":"f82ad008-d4ee-471f-bf5d-7f5849645684","Type":"ContainerDied","Data":"a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c"}
Jul 08 16:54:17.678610 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.678519    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"
Jul 08 16:54:17.678610 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.678580    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v" event={"ID":"f82ad008-d4ee-471f-bf5d-7f5849645684","Type":"ContainerDied","Data":"9a744fb90d42bbbae16d12507ac91de76b7ecb0285e558f020ee7cea20e06b21"}
Jul 08 16:54:17.678732 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.678613    2550 scope.go:117] "RemoveContainer" containerID="85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65"
Jul 08 16:54:17.679554 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.679454    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f82ad008-d4ee-471f-bf5d-7f5849645684-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:17.679554 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.679471    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f82ad008-d4ee-471f-bf5d-7f5849645684-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:17.679554 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.679484    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwx8x\" (UniqueName: \"kubernetes.io/projected/f82ad008-d4ee-471f-bf5d-7f5849645684-kube-api-access-cwx8x\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:17.679554 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.679497    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f82ad008-d4ee-471f-bf5d-7f5849645684-cabundle-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:17.679554 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.679509    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f82ad008-d4ee-471f-bf5d-7f5849645684-kserve-provision-location\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:17.687255 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.687241    2550 scope.go:117] "RemoveContainer" containerID="a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c"
Jul 08 16:54:17.694085 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.694068    2550 scope.go:117] "RemoveContainer" containerID="f8ca889206c7a7d6ea2e7ed9d962a337bdea4201879a72fa73e9a584405bf99b"
Jul 08 16:54:17.700598 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.700583    2550 scope.go:117] "RemoveContainer" containerID="85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65"
Jul 08 16:54:17.700847 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:54:17.700826    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65\": container with ID starting with 85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65 not found: ID does not exist" containerID="85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65"
Jul 08 16:54:17.700936 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.700851    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65"} err="failed to get container status \"85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65\": rpc error: code = NotFound desc = could not find container \"85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65\": container with ID starting with 85f94ed3f5dd1f7f2db24263760d4ec9274121e23d16f84f56d32c346fabce65 not found: ID does not exist"
Jul 08 16:54:17.700936 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.700867    2550 scope.go:117] "RemoveContainer" containerID="a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c"
Jul 08 16:54:17.701092 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:54:17.701066    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c\": container with ID starting with a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c not found: ID does not exist" containerID="a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c"
Jul 08 16:54:17.701137 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.701095    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c"} err="failed to get container status \"a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c\": rpc error: code = NotFound desc = could not find container \"a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c\": container with ID starting with a0e54925be541782033cadc743439fe4fabeea07cc7781ab1466af7e3399767c not found: ID does not exist"
Jul 08 16:54:17.701137 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.701108    2550 scope.go:117] "RemoveContainer" containerID="f8ca889206c7a7d6ea2e7ed9d962a337bdea4201879a72fa73e9a584405bf99b"
Jul 08 16:54:17.701287 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:54:17.701268    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ca889206c7a7d6ea2e7ed9d962a337bdea4201879a72fa73e9a584405bf99b\": container with ID starting with f8ca889206c7a7d6ea2e7ed9d962a337bdea4201879a72fa73e9a584405bf99b not found: ID does not exist" containerID="f8ca889206c7a7d6ea2e7ed9d962a337bdea4201879a72fa73e9a584405bf99b"
Jul 08 16:54:17.701334 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.701295    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ca889206c7a7d6ea2e7ed9d962a337bdea4201879a72fa73e9a584405bf99b"} err="failed to get container status \"f8ca889206c7a7d6ea2e7ed9d962a337bdea4201879a72fa73e9a584405bf99b\": rpc error: code = NotFound desc = could not find container \"f8ca889206c7a7d6ea2e7ed9d962a337bdea4201879a72fa73e9a584405bf99b\": container with ID starting with f8ca889206c7a7d6ea2e7ed9d962a337bdea4201879a72fa73e9a584405bf99b not found: ID does not exist"
Jul 08 16:54:17.712553 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.712531    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"]
Jul 08 16:54:17.715862 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:17.715842    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b29bb6-predictor-99c6d7f9b-bsr8v"]
Jul 08 16:54:18.833677 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.833643    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"]
Jul 08 16:54:18.834103 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834028    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" containerName="storage-initializer"
Jul 08 16:54:18.834103 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834045    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" containerName="storage-initializer"
Jul 08 16:54:18.834103 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834057    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kube-rbac-proxy"
Jul 08 16:54:18.834103 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834065    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kube-rbac-proxy"
Jul 08 16:54:18.834103 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834084    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" containerName="storage-initializer"
Jul 08 16:54:18.834103 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834093    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" containerName="storage-initializer"
Jul 08 16:54:18.834103 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834105    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kserve-container"
Jul 08 16:54:18.834449 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834112    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kserve-container"
Jul 08 16:54:18.834449 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834133    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="storage-initializer"
Jul 08 16:54:18.834449 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834142    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="storage-initializer"
Jul 08 16:54:18.834449 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834214    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" containerName="storage-initializer"
Jul 08 16:54:18.834449 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834224    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecaee57b-2a3c-4351-bc3e-2e4e1c99652f" containerName="storage-initializer"
Jul 08 16:54:18.834449 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834236    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kube-rbac-proxy"
Jul 08 16:54:18.834449 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.834246    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" containerName="kserve-container"
Jul 08 16:54:18.837178 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.837158    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:18.842370 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.842345    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:54:18.842700 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.842682    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-b37da8-predictor-serving-cert\""
Jul 08 16:54:18.843058 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.843039    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-b37da8-kube-rbac-proxy-sar-config\""
Jul 08 16:54:18.843058 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.843049    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:54:18.843175 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.843042    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-b37da8\""
Jul 08 16:54:18.843265 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.843247    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-b37da8-dockercfg-frmrk\""
Jul 08 16:54:18.843323 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.843303    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:54:18.853396 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.853378    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"]
Jul 08 16:54:18.988707 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.988683    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df7cedce-f644-401f-9b2e-96bc86d42365-proxy-tls\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:18.988813 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.988709    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df7cedce-f644-401f-9b2e-96bc86d42365-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:18.988813 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.988734    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd9df\" (UniqueName: \"kubernetes.io/projected/df7cedce-f644-401f-9b2e-96bc86d42365-kube-api-access-sd9df\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:18.988934 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.988813    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df7cedce-f644-401f-9b2e-96bc86d42365-kserve-provision-location\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:18.988934 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:18.988885    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df7cedce-f644-401f-9b2e-96bc86d42365-cabundle-cert\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:19.027526 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.027503    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f82ad008-d4ee-471f-bf5d-7f5849645684" path="/var/lib/kubelet/pods/f82ad008-d4ee-471f-bf5d-7f5849645684/volumes"
Jul 08 16:54:19.090116 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.090070    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df7cedce-f644-401f-9b2e-96bc86d42365-kserve-provision-location\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:19.090116 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.090106    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df7cedce-f644-401f-9b2e-96bc86d42365-cabundle-cert\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:19.090430 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.090124    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df7cedce-f644-401f-9b2e-96bc86d42365-proxy-tls\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:19.090430 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.090139    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df7cedce-f644-401f-9b2e-96bc86d42365-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:19.090430 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.090172    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9df\" (UniqueName: \"kubernetes.io/projected/df7cedce-f644-401f-9b2e-96bc86d42365-kube-api-access-sd9df\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:19.090612 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.090443    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df7cedce-f644-401f-9b2e-96bc86d42365-kserve-provision-location\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:19.090831 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.090809    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df7cedce-f644-401f-9b2e-96bc86d42365-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:19.090882 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.090814    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df7cedce-f644-401f-9b2e-96bc86d42365-cabundle-cert\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:19.092654 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.092636    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df7cedce-f644-401f-9b2e-96bc86d42365-proxy-tls\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:19.104550 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.104519    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9df\" (UniqueName: \"kubernetes.io/projected/df7cedce-f644-401f-9b2e-96bc86d42365-kube-api-access-sd9df\") pod \"isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") " pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:19.152254 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.152222    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:19.286039 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.286020    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"]
Jul 08 16:54:19.288466 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:54:19.288440    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7cedce_f644_401f_9b2e_96bc86d42365.slice/crio-ca370ce792068bcc8a7d020787557940ab7e3156cc8edae6e86976ad8e1bb1a1 WatchSource:0}: Error finding container ca370ce792068bcc8a7d020787557940ab7e3156cc8edae6e86976ad8e1bb1a1: Status 404 returned error can't find the container with id ca370ce792068bcc8a7d020787557940ab7e3156cc8edae6e86976ad8e1bb1a1
Jul 08 16:54:19.689693 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.689592    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh" event={"ID":"df7cedce-f644-401f-9b2e-96bc86d42365","Type":"ContainerStarted","Data":"30aab8f9dc10f8dd3f69b9c149fff050f63aa89f66e1bc5eec75e554a9246a6a"}
Jul 08 16:54:19.689693 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:19.689675    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh" event={"ID":"df7cedce-f644-401f-9b2e-96bc86d42365","Type":"ContainerStarted","Data":"ca370ce792068bcc8a7d020787557940ab7e3156cc8edae6e86976ad8e1bb1a1"}
Jul 08 16:54:24.711237 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:24.711202    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh_df7cedce-f644-401f-9b2e-96bc86d42365/storage-initializer/0.log"
Jul 08 16:54:24.711653 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:24.711252    2550 generic.go:356] "Generic (PLEG): container finished" podID="df7cedce-f644-401f-9b2e-96bc86d42365" containerID="30aab8f9dc10f8dd3f69b9c149fff050f63aa89f66e1bc5eec75e554a9246a6a" exitCode=1
Jul 08 16:54:24.711653 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:24.711324    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh" event={"ID":"df7cedce-f644-401f-9b2e-96bc86d42365","Type":"ContainerDied","Data":"30aab8f9dc10f8dd3f69b9c149fff050f63aa89f66e1bc5eec75e554a9246a6a"}
Jul 08 16:54:25.716995 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:25.716967    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh_df7cedce-f644-401f-9b2e-96bc86d42365/storage-initializer/0.log"
Jul 08 16:54:25.717340 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:25.717025    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh" event={"ID":"df7cedce-f644-401f-9b2e-96bc86d42365","Type":"ContainerStarted","Data":"8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355"}
Jul 08 16:54:28.827806 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:28.827778    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"]
Jul 08 16:54:28.828193 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:28.827983    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh" podUID="df7cedce-f644-401f-9b2e-96bc86d42365" containerName="storage-initializer" containerID="cri-o://8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355" gracePeriod=30
Jul 08 16:54:30.559150 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.559130    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh_df7cedce-f644-401f-9b2e-96bc86d42365/storage-initializer/1.log"
Jul 08 16:54:30.559529 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.559515    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh_df7cedce-f644-401f-9b2e-96bc86d42365/storage-initializer/0.log"
Jul 08 16:54:30.561313 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.561298    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:30.572872 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.572847    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df7cedce-f644-401f-9b2e-96bc86d42365-kube-rbac-proxy-sar-config\") pod \"df7cedce-f644-401f-9b2e-96bc86d42365\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") "
Jul 08 16:54:30.572978 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.572876    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd9df\" (UniqueName: \"kubernetes.io/projected/df7cedce-f644-401f-9b2e-96bc86d42365-kube-api-access-sd9df\") pod \"df7cedce-f644-401f-9b2e-96bc86d42365\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") "
Jul 08 16:54:30.572978 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.572893    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df7cedce-f644-401f-9b2e-96bc86d42365-kserve-provision-location\") pod \"df7cedce-f644-401f-9b2e-96bc86d42365\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") "
Jul 08 16:54:30.572978 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.572917    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df7cedce-f644-401f-9b2e-96bc86d42365-cabundle-cert\") pod \"df7cedce-f644-401f-9b2e-96bc86d42365\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") "
Jul 08 16:54:30.572978 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.572944    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df7cedce-f644-401f-9b2e-96bc86d42365-proxy-tls\") pod \"df7cedce-f644-401f-9b2e-96bc86d42365\" (UID: \"df7cedce-f644-401f-9b2e-96bc86d42365\") "
Jul 08 16:54:30.573205 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.573162    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7cedce-f644-401f-9b2e-96bc86d42365-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df7cedce-f644-401f-9b2e-96bc86d42365" (UID: "df7cedce-f644-401f-9b2e-96bc86d42365"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:54:30.573299 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.573276    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7cedce-f644-401f-9b2e-96bc86d42365-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df7cedce-f644-401f-9b2e-96bc86d42365" (UID: "df7cedce-f644-401f-9b2e-96bc86d42365"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:54:30.573368 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.573346    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7cedce-f644-401f-9b2e-96bc86d42365-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df7cedce-f644-401f-9b2e-96bc86d42365" (UID: "df7cedce-f644-401f-9b2e-96bc86d42365"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:54:30.575047 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.575019    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7cedce-f644-401f-9b2e-96bc86d42365-kube-api-access-sd9df" (OuterVolumeSpecName: "kube-api-access-sd9df") pod "df7cedce-f644-401f-9b2e-96bc86d42365" (UID: "df7cedce-f644-401f-9b2e-96bc86d42365"). InnerVolumeSpecName "kube-api-access-sd9df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:54:30.575138 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.575059    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df7cedce-f644-401f-9b2e-96bc86d42365-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df7cedce-f644-401f-9b2e-96bc86d42365" (UID: "df7cedce-f644-401f-9b2e-96bc86d42365"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:54:30.673714 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.673665    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df7cedce-f644-401f-9b2e-96bc86d42365-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:30.673714 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.673683    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df7cedce-f644-401f-9b2e-96bc86d42365-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:30.673714 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.673693    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd9df\" (UniqueName: \"kubernetes.io/projected/df7cedce-f644-401f-9b2e-96bc86d42365-kube-api-access-sd9df\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:30.673714 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.673702    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df7cedce-f644-401f-9b2e-96bc86d42365-kserve-provision-location\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:30.673714 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.673710    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df7cedce-f644-401f-9b2e-96bc86d42365-cabundle-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:54:30.735719 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.735702    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh_df7cedce-f644-401f-9b2e-96bc86d42365/storage-initializer/1.log"
Jul 08 16:54:30.736083 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.736070    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh_df7cedce-f644-401f-9b2e-96bc86d42365/storage-initializer/0.log"
Jul 08 16:54:30.736151 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.736103    2550 generic.go:356] "Generic (PLEG): container finished" podID="df7cedce-f644-401f-9b2e-96bc86d42365" containerID="8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355" exitCode=1
Jul 08 16:54:30.736151 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.736131    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh" event={"ID":"df7cedce-f644-401f-9b2e-96bc86d42365","Type":"ContainerDied","Data":"8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355"}
Jul 08 16:54:30.736218 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.736162    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"
Jul 08 16:54:30.736218 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.736211    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh" event={"ID":"df7cedce-f644-401f-9b2e-96bc86d42365","Type":"ContainerDied","Data":"ca370ce792068bcc8a7d020787557940ab7e3156cc8edae6e86976ad8e1bb1a1"}
Jul 08 16:54:30.736279 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.736237    2550 scope.go:117] "RemoveContainer" containerID="8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355"
Jul 08 16:54:30.744369 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.744355    2550 scope.go:117] "RemoveContainer" containerID="30aab8f9dc10f8dd3f69b9c149fff050f63aa89f66e1bc5eec75e554a9246a6a"
Jul 08 16:54:30.751182 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.751165    2550 scope.go:117] "RemoveContainer" containerID="8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355"
Jul 08 16:54:30.751415 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:54:30.751399    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355\": container with ID starting with 8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355 not found: ID does not exist" containerID="8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355"
Jul 08 16:54:30.751480 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.751426    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355"} err="failed to get container status \"8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355\": rpc error: code = NotFound desc = could not find container \"8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355\": container with ID starting with 8cc91cf4b85c445c73c66fe2c509181d9285332cf643721723163fabbb2f2355 not found: ID does not exist"
Jul 08 16:54:30.751480 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.751450    2550 scope.go:117] "RemoveContainer" containerID="30aab8f9dc10f8dd3f69b9c149fff050f63aa89f66e1bc5eec75e554a9246a6a"
Jul 08 16:54:30.751718 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:54:30.751701    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30aab8f9dc10f8dd3f69b9c149fff050f63aa89f66e1bc5eec75e554a9246a6a\": container with ID starting with 30aab8f9dc10f8dd3f69b9c149fff050f63aa89f66e1bc5eec75e554a9246a6a not found: ID does not exist" containerID="30aab8f9dc10f8dd3f69b9c149fff050f63aa89f66e1bc5eec75e554a9246a6a"
Jul 08 16:54:30.751756 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.751723    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30aab8f9dc10f8dd3f69b9c149fff050f63aa89f66e1bc5eec75e554a9246a6a"} err="failed to get container status \"30aab8f9dc10f8dd3f69b9c149fff050f63aa89f66e1bc5eec75e554a9246a6a\": rpc error: code = NotFound desc = could not find container \"30aab8f9dc10f8dd3f69b9c149fff050f63aa89f66e1bc5eec75e554a9246a6a\": container with ID starting with 30aab8f9dc10f8dd3f69b9c149fff050f63aa89f66e1bc5eec75e554a9246a6a not found: ID does not exist"
Jul 08 16:54:30.771788 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.771769    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"]
Jul 08 16:54:30.778055 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:30.778038    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b37da8-predictor-76d7555bd7-7wmgh"]
Jul 08 16:54:31.027257 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:31.027204    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df7cedce-f644-401f-9b2e-96bc86d42365" path="/var/lib/kubelet/pods/df7cedce-f644-401f-9b2e-96bc86d42365/volumes"
Jul 08 16:54:34.049389 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.049354    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"]
Jul 08 16:54:34.049809 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.049770    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df7cedce-f644-401f-9b2e-96bc86d42365" containerName="storage-initializer"
Jul 08 16:54:34.049809 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.049787    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="df7cedce-f644-401f-9b2e-96bc86d42365" containerName="storage-initializer"
Jul 08 16:54:34.049932 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.049888    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="df7cedce-f644-401f-9b2e-96bc86d42365" containerName="storage-initializer"
Jul 08 16:54:34.049932 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.049899    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="df7cedce-f644-401f-9b2e-96bc86d42365" containerName="storage-initializer"
Jul 08 16:54:34.050030 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.049980    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df7cedce-f644-401f-9b2e-96bc86d42365" containerName="storage-initializer"
Jul 08 16:54:34.050030 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.049990    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="df7cedce-f644-401f-9b2e-96bc86d42365" containerName="storage-initializer"
Jul 08 16:54:34.053866 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.053846    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.056439 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.056415    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:54:34.056439 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.056432    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-6e855-predictor-serving-cert\""
Jul 08 16:54:34.056618 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.056419    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:54:34.057522 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.057505    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:54:34.057658 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.057641    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-6e855-kube-rbac-proxy-sar-config\""
Jul 08 16:54:34.057658 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.057642    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7vdnq\""
Jul 08 16:54:34.057796 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.057712    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:54:34.065301 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.065282    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"]
Jul 08 16:54:34.094292 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.094265    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3da0e886-84bd-4b9a-b67b-2c30f3771246-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.094393 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.094296    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3da0e886-84bd-4b9a-b67b-2c30f3771246-cabundle-cert\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.094393 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.094322    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rlpb\" (UniqueName: \"kubernetes.io/projected/3da0e886-84bd-4b9a-b67b-2c30f3771246-kube-api-access-7rlpb\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.094393 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.094366    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3da0e886-84bd-4b9a-b67b-2c30f3771246-proxy-tls\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.094520 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.094404    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3da0e886-84bd-4b9a-b67b-2c30f3771246-kserve-provision-location\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.195408 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.195387    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rlpb\" (UniqueName: \"kubernetes.io/projected/3da0e886-84bd-4b9a-b67b-2c30f3771246-kube-api-access-7rlpb\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.195490 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.195416    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3da0e886-84bd-4b9a-b67b-2c30f3771246-proxy-tls\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.195490 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.195441    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3da0e886-84bd-4b9a-b67b-2c30f3771246-kserve-provision-location\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.195603 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.195487    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3da0e886-84bd-4b9a-b67b-2c30f3771246-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.195603 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.195523    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3da0e886-84bd-4b9a-b67b-2c30f3771246-cabundle-cert\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.195876 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.195857    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3da0e886-84bd-4b9a-b67b-2c30f3771246-kserve-provision-location\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.196175 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.196151    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3da0e886-84bd-4b9a-b67b-2c30f3771246-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.196248 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.196161    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3da0e886-84bd-4b9a-b67b-2c30f3771246-cabundle-cert\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.197790 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.197771    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3da0e886-84bd-4b9a-b67b-2c30f3771246-proxy-tls\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.204720 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.204700    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rlpb\" (UniqueName: \"kubernetes.io/projected/3da0e886-84bd-4b9a-b67b-2c30f3771246-kube-api-access-7rlpb\") pod \"raw-sklearn-6e855-predictor-67c4c4b6c7-q6828\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") " pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.363589 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.363513    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:34.493402 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.493282    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"]
Jul 08 16:54:34.496272 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:54:34.496236    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da0e886_84bd_4b9a_b67b_2c30f3771246.slice/crio-224ef8790d370f2888138351da57cee69150b4e73c1b736aae83e8ba6de18cf1 WatchSource:0}: Error finding container 224ef8790d370f2888138351da57cee69150b4e73c1b736aae83e8ba6de18cf1: Status 404 returned error can't find the container with id 224ef8790d370f2888138351da57cee69150b4e73c1b736aae83e8ba6de18cf1
Jul 08 16:54:34.754237 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.754162    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" event={"ID":"3da0e886-84bd-4b9a-b67b-2c30f3771246","Type":"ContainerStarted","Data":"c528d49bcb2099dda91eb3b388681c940137c17a38fa5f4a4ff67964c24b5cda"}
Jul 08 16:54:34.754369 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:34.754244    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" event={"ID":"3da0e886-84bd-4b9a-b67b-2c30f3771246","Type":"ContainerStarted","Data":"224ef8790d370f2888138351da57cee69150b4e73c1b736aae83e8ba6de18cf1"}
Jul 08 16:54:38.770088 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:38.770058    2550 generic.go:356] "Generic (PLEG): container finished" podID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerID="c528d49bcb2099dda91eb3b388681c940137c17a38fa5f4a4ff67964c24b5cda" exitCode=0
Jul 08 16:54:38.770425 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:38.770133    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" event={"ID":"3da0e886-84bd-4b9a-b67b-2c30f3771246","Type":"ContainerDied","Data":"c528d49bcb2099dda91eb3b388681c940137c17a38fa5f4a4ff67964c24b5cda"}
Jul 08 16:54:39.780970 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:39.780938    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" event={"ID":"3da0e886-84bd-4b9a-b67b-2c30f3771246","Type":"ContainerStarted","Data":"65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4"}
Jul 08 16:54:39.781338 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:39.781028    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" event={"ID":"3da0e886-84bd-4b9a-b67b-2c30f3771246","Type":"ContainerStarted","Data":"646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2"}
Jul 08 16:54:39.781338 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:39.781328    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:39.801144 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:39.801106    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podStartSLOduration=2.304286267 podStartE2EDuration="5.801092381s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.496806114s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:54:39.800513974 +0000 UTC m=+1195.337066459" watchObservedRunningTime="2026-07-08 16:54:39.801092381 +0000 UTC m=+1195.337644865"
Jul 08 16:54:40.784627 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:40.784594    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:40.785684 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:40.785660    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:54:41.788827 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:41.788785    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:54:46.793255 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:46.793225    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:54:46.793892 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:46.793863    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:54:56.794623 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:54:56.794559    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:55:06.793859 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:06.793825    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:55:16.794318 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:16.794280    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:55:26.794024 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:26.793984    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:55:36.794282 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:36.794198    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:55:46.795185 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:46.795157    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:55:54.141966 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:54.141932    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"]
Jul 08 16:55:54.142341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:54.142156    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container" containerID="cri-o://646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2" gracePeriod=30
Jul 08 16:55:54.142341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:54.142176    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kube-rbac-proxy" containerID="cri-o://65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4" gracePeriod=30
Jul 08 16:55:55.063886 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:55.063850    2550 generic.go:356] "Generic (PLEG): container finished" podID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerID="65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4" exitCode=2
Jul 08 16:55:55.064055 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:55.063924    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" event={"ID":"3da0e886-84bd-4b9a-b67b-2c30f3771246","Type":"ContainerDied","Data":"65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4"}
Jul 08 16:55:56.789749 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:56.789703    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 08 16:55:56.794092 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:56.794057    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 08 16:55:58.383554 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.383535    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:55:58.495078 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.494997    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3da0e886-84bd-4b9a-b67b-2c30f3771246-kube-rbac-proxy-sar-config\") pod \"3da0e886-84bd-4b9a-b67b-2c30f3771246\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") "
Jul 08 16:55:58.495078 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.495033    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rlpb\" (UniqueName: \"kubernetes.io/projected/3da0e886-84bd-4b9a-b67b-2c30f3771246-kube-api-access-7rlpb\") pod \"3da0e886-84bd-4b9a-b67b-2c30f3771246\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") "
Jul 08 16:55:58.495078 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.495059    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3da0e886-84bd-4b9a-b67b-2c30f3771246-kserve-provision-location\") pod \"3da0e886-84bd-4b9a-b67b-2c30f3771246\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") "
Jul 08 16:55:58.495078 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.495081    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3da0e886-84bd-4b9a-b67b-2c30f3771246-proxy-tls\") pod \"3da0e886-84bd-4b9a-b67b-2c30f3771246\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") "
Jul 08 16:55:58.495396 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.495127    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3da0e886-84bd-4b9a-b67b-2c30f3771246-cabundle-cert\") pod \"3da0e886-84bd-4b9a-b67b-2c30f3771246\" (UID: \"3da0e886-84bd-4b9a-b67b-2c30f3771246\") "
Jul 08 16:55:58.495452 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.495420    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da0e886-84bd-4b9a-b67b-2c30f3771246-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3da0e886-84bd-4b9a-b67b-2c30f3771246" (UID: "3da0e886-84bd-4b9a-b67b-2c30f3771246"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:55:58.495452 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.495442    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da0e886-84bd-4b9a-b67b-2c30f3771246-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3da0e886-84bd-4b9a-b67b-2c30f3771246" (UID: "3da0e886-84bd-4b9a-b67b-2c30f3771246"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:55:58.495593 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.495545    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da0e886-84bd-4b9a-b67b-2c30f3771246-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3da0e886-84bd-4b9a-b67b-2c30f3771246" (UID: "3da0e886-84bd-4b9a-b67b-2c30f3771246"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:55:58.497300 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.497280    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da0e886-84bd-4b9a-b67b-2c30f3771246-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3da0e886-84bd-4b9a-b67b-2c30f3771246" (UID: "3da0e886-84bd-4b9a-b67b-2c30f3771246"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:55:58.497368 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.497347    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da0e886-84bd-4b9a-b67b-2c30f3771246-kube-api-access-7rlpb" (OuterVolumeSpecName: "kube-api-access-7rlpb") pod "3da0e886-84bd-4b9a-b67b-2c30f3771246" (UID: "3da0e886-84bd-4b9a-b67b-2c30f3771246"). InnerVolumeSpecName "kube-api-access-7rlpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:55:58.595897 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.595875    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3da0e886-84bd-4b9a-b67b-2c30f3771246-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:55:58.595897 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.595894    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rlpb\" (UniqueName: \"kubernetes.io/projected/3da0e886-84bd-4b9a-b67b-2c30f3771246-kube-api-access-7rlpb\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:55:58.596016 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.595903    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3da0e886-84bd-4b9a-b67b-2c30f3771246-kserve-provision-location\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:55:58.596016 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.595912    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3da0e886-84bd-4b9a-b67b-2c30f3771246-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:55:58.596016 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:58.595920    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3da0e886-84bd-4b9a-b67b-2c30f3771246-cabundle-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:55:59.080090 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.080058    2550 generic.go:356] "Generic (PLEG): container finished" podID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerID="646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2" exitCode=0
Jul 08 16:55:59.080201 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.080106    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"
Jul 08 16:55:59.080201 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.080110    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" event={"ID":"3da0e886-84bd-4b9a-b67b-2c30f3771246","Type":"ContainerDied","Data":"646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2"}
Jul 08 16:55:59.080201 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.080163    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828" event={"ID":"3da0e886-84bd-4b9a-b67b-2c30f3771246","Type":"ContainerDied","Data":"224ef8790d370f2888138351da57cee69150b4e73c1b736aae83e8ba6de18cf1"}
Jul 08 16:55:59.080304 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.080189    2550 scope.go:117] "RemoveContainer" containerID="65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4"
Jul 08 16:55:59.088014 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.087995    2550 scope.go:117] "RemoveContainer" containerID="646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2"
Jul 08 16:55:59.094906 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.094890    2550 scope.go:117] "RemoveContainer" containerID="c528d49bcb2099dda91eb3b388681c940137c17a38fa5f4a4ff67964c24b5cda"
Jul 08 16:55:59.097265 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.097246    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"]
Jul 08 16:55:59.102381 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.102288    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-6e855-predictor-67c4c4b6c7-q6828"]
Jul 08 16:55:59.102452 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.102404    2550 scope.go:117] "RemoveContainer" containerID="65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4"
Jul 08 16:55:59.102710 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:55:59.102689    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4\": container with ID starting with 65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4 not found: ID does not exist" containerID="65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4"
Jul 08 16:55:59.102763 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.102718    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4"} err="failed to get container status \"65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4\": rpc error: code = NotFound desc = could not find container \"65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4\": container with ID starting with 65ba78578b2367a3e29bde3a3873c4aef05ab14aac3beffa9d11383a0347f5a4 not found: ID does not exist"
Jul 08 16:55:59.102763 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.102734    2550 scope.go:117] "RemoveContainer" containerID="646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2"
Jul 08 16:55:59.102950 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:55:59.102921    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2\": container with ID starting with 646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2 not found: ID does not exist" containerID="646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2"
Jul 08 16:55:59.103015 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.102962    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2"} err="failed to get container status \"646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2\": rpc error: code = NotFound desc = could not find container \"646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2\": container with ID starting with 646e3c9fab5a7502967afd1ca95fd53297fb1e73029847fe2bd7c6f6288fe6b2 not found: ID does not exist"
Jul 08 16:55:59.103015 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.102986    2550 scope.go:117] "RemoveContainer" containerID="c528d49bcb2099dda91eb3b388681c940137c17a38fa5f4a4ff67964c24b5cda"
Jul 08 16:55:59.103227 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:55:59.103204    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c528d49bcb2099dda91eb3b388681c940137c17a38fa5f4a4ff67964c24b5cda\": container with ID starting with c528d49bcb2099dda91eb3b388681c940137c17a38fa5f4a4ff67964c24b5cda not found: ID does not exist" containerID="c528d49bcb2099dda91eb3b388681c940137c17a38fa5f4a4ff67964c24b5cda"
Jul 08 16:55:59.103227 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.103222    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c528d49bcb2099dda91eb3b388681c940137c17a38fa5f4a4ff67964c24b5cda"} err="failed to get container status \"c528d49bcb2099dda91eb3b388681c940137c17a38fa5f4a4ff67964c24b5cda\": rpc error: code = NotFound desc = could not find container \"c528d49bcb2099dda91eb3b388681c940137c17a38fa5f4a4ff67964c24b5cda\": container with ID starting with c528d49bcb2099dda91eb3b388681c940137c17a38fa5f4a4ff67964c24b5cda not found: ID does not exist"
Jul 08 16:55:59.321189 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.321111    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"]
Jul 08 16:55:59.321392 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.321380    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container"
Jul 08 16:55:59.321455 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.321392    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container"
Jul 08 16:55:59.321455 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.321399    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kube-rbac-proxy"
Jul 08 16:55:59.321455 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.321405    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kube-rbac-proxy"
Jul 08 16:55:59.321455 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.321413    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="storage-initializer"
Jul 08 16:55:59.321455 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.321419    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="storage-initializer"
Jul 08 16:55:59.321728 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.321470    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kserve-container"
Jul 08 16:55:59.321728 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.321478    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" containerName="kube-rbac-proxy"
Jul 08 16:55:59.325864 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.325846    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.328434 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.328411    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:55:59.328434 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.328426    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-7d1b7-predictor-serving-cert\""
Jul 08 16:55:59.328634 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.328443    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:55:59.328709 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.328696    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7vdnq\""
Jul 08 16:55:59.329062 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.329044    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:55:59.329147 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.329071    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-7d1b7-kube-rbac-proxy-sar-config\""
Jul 08 16:55:59.329147 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.329080    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:55:59.335420 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.335401    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"]
Jul 08 16:55:59.403608 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.403589    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kserve-provision-location\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.403905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.403614    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.403905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.403640    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-cabundle-cert\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.403905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.403720    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hhxf\" (UniqueName: \"kubernetes.io/projected/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kube-api-access-8hhxf\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.403905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.403769    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-proxy-tls\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.504242 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.504220    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hhxf\" (UniqueName: \"kubernetes.io/projected/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kube-api-access-8hhxf\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.504334 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.504260    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-proxy-tls\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.504334 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.504285    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kserve-provision-location\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.504334 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.504301    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.504334 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.504324    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-cabundle-cert\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.504694 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.504677    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kserve-provision-location\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.505002 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.504983    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-cabundle-cert\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.505002 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.504997    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.506593 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.506577    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-proxy-tls\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.514512 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.514493    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hhxf\" (UniqueName: \"kubernetes.io/projected/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kube-api-access-8hhxf\") pod \"raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.636846 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.636795    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:55:59.765309 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:55:59.765279    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"]
Jul 08 16:55:59.768155 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:55:59.768130    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6cdd1f_b5d8_4884_b60f_8e10519ee00f.slice/crio-c1f2cf6e013e28bea0bdb580735dd4090c5a53ca83f63c5ad7a962a2d1946258 WatchSource:0}: Error finding container c1f2cf6e013e28bea0bdb580735dd4090c5a53ca83f63c5ad7a962a2d1946258: Status 404 returned error can't find the container with id c1f2cf6e013e28bea0bdb580735dd4090c5a53ca83f63c5ad7a962a2d1946258
Jul 08 16:56:00.091055 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:00.091007    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" event={"ID":"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f","Type":"ContainerStarted","Data":"1337368cd9b095c007500288e40a38923308d71504a81eccdc589228595f6aea"}
Jul 08 16:56:00.091216 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:00.091113    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" event={"ID":"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f","Type":"ContainerStarted","Data":"c1f2cf6e013e28bea0bdb580735dd4090c5a53ca83f63c5ad7a962a2d1946258"}
Jul 08 16:56:01.027732 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:01.027701    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3da0e886-84bd-4b9a-b67b-2c30f3771246" path="/var/lib/kubelet/pods/3da0e886-84bd-4b9a-b67b-2c30f3771246/volumes"
Jul 08 16:56:04.108930 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:04.108899    2550 generic.go:356] "Generic (PLEG): container finished" podID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerID="1337368cd9b095c007500288e40a38923308d71504a81eccdc589228595f6aea" exitCode=0
Jul 08 16:56:04.109257 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:04.108974    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" event={"ID":"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f","Type":"ContainerDied","Data":"1337368cd9b095c007500288e40a38923308d71504a81eccdc589228595f6aea"}
Jul 08 16:56:05.114627 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:05.114592    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" event={"ID":"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f","Type":"ContainerStarted","Data":"6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437"}
Jul 08 16:56:05.114974 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:05.114659    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" event={"ID":"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f","Type":"ContainerStarted","Data":"33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb"}
Jul 08 16:56:05.114974 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:05.114842    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:56:05.133896 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:05.133858    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podStartSLOduration=2.630952801 podStartE2EDuration="6.133846283s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.502893482s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:56:05.133152951 +0000 UTC m=+1280.669705438" watchObservedRunningTime="2026-07-08 16:56:05.133846283 +0000 UTC m=+1280.670398802"
Jul 08 16:56:06.119266 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:06.119236    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:56:06.120418 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:06.120390    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:56:07.122908 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:07.122867    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:56:12.127620 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:12.127590    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:56:12.128034 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:12.128009    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:56:22.128775 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:22.128733    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:56:32.128443 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:32.128403    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:56:42.128875 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:42.128832    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:56:52.128986 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:56:52.128949    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:57:02.128329 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:02.128295    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:57:12.129342 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:12.129277    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:57:19.418474 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:19.418442    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"]
Jul 08 16:57:19.418864 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:19.418689    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container" containerID="cri-o://33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb" gracePeriod=30
Jul 08 16:57:19.418864 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:19.418740    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kube-rbac-proxy" containerID="cri-o://6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437" gracePeriod=30
Jul 08 16:57:20.391034 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:20.391004    2550 generic.go:356] "Generic (PLEG): container finished" podID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerID="6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437" exitCode=2
Jul 08 16:57:20.391204 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:20.391077    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" event={"ID":"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f","Type":"ContainerDied","Data":"6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437"}
Jul 08 16:57:22.123325 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.123279    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 08 16:57:22.128776 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.128750    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 16:57:22.964459 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.964437    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:57:22.997348 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.997323    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hhxf\" (UniqueName: \"kubernetes.io/projected/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kube-api-access-8hhxf\") pod \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") "
Jul 08 16:57:22.997509 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.997358    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kube-rbac-proxy-sar-config\") pod \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") "
Jul 08 16:57:22.997509 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.997395    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kserve-provision-location\") pod \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") "
Jul 08 16:57:22.997509 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.997461    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-cabundle-cert\") pod \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") "
Jul 08 16:57:22.997509 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.997492    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-proxy-tls\") pod \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\" (UID: \"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f\") "
Jul 08 16:57:22.997814 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.997776    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" (UID: "3d6cdd1f-b5d8-4884-b60f-8e10519ee00f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:57:22.997814 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.997800    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" (UID: "3d6cdd1f-b5d8-4884-b60f-8e10519ee00f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:57:22.997924 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.997842    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" (UID: "3d6cdd1f-b5d8-4884-b60f-8e10519ee00f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:57:22.999638 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.999612    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kube-api-access-8hhxf" (OuterVolumeSpecName: "kube-api-access-8hhxf") pod "3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" (UID: "3d6cdd1f-b5d8-4884-b60f-8e10519ee00f"). InnerVolumeSpecName "kube-api-access-8hhxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:57:22.999849 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:22.999831    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" (UID: "3d6cdd1f-b5d8-4884-b60f-8e10519ee00f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:57:23.098583 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.098499    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kserve-provision-location\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:57:23.098583 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.098525    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-cabundle-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:57:23.098583 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.098536    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-proxy-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:57:23.098583 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.098544    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hhxf\" (UniqueName: \"kubernetes.io/projected/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kube-api-access-8hhxf\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:57:23.098583 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.098551    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:57:23.403475 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.403395    2550 generic.go:356] "Generic (PLEG): container finished" podID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerID="33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb" exitCode=0
Jul 08 16:57:23.403475 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.403454    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"
Jul 08 16:57:23.404079 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.403476    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" event={"ID":"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f","Type":"ContainerDied","Data":"33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb"}
Jul 08 16:57:23.404079 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.403554    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5" event={"ID":"3d6cdd1f-b5d8-4884-b60f-8e10519ee00f","Type":"ContainerDied","Data":"c1f2cf6e013e28bea0bdb580735dd4090c5a53ca83f63c5ad7a962a2d1946258"}
Jul 08 16:57:23.404079 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.403594    2550 scope.go:117] "RemoveContainer" containerID="6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437"
Jul 08 16:57:23.411519 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.411503    2550 scope.go:117] "RemoveContainer" containerID="33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb"
Jul 08 16:57:23.418640 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.418619    2550 scope.go:117] "RemoveContainer" containerID="1337368cd9b095c007500288e40a38923308d71504a81eccdc589228595f6aea"
Jul 08 16:57:23.420215 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.420194    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"]
Jul 08 16:57:23.425238 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.425218    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7d1b7-predictor-698dfdb665-krtq5"]
Jul 08 16:57:23.425979 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.425965    2550 scope.go:117] "RemoveContainer" containerID="6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437"
Jul 08 16:57:23.426206 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:57:23.426188    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437\": container with ID starting with 6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437 not found: ID does not exist" containerID="6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437"
Jul 08 16:57:23.426276 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.426219    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437"} err="failed to get container status \"6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437\": rpc error: code = NotFound desc = could not find container \"6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437\": container with ID starting with 6729b4068b47f2a826563ef54873e3aa6b13e0cfba47004d60b9987fb9ef6437 not found: ID does not exist"
Jul 08 16:57:23.426276 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.426241    2550 scope.go:117] "RemoveContainer" containerID="33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb"
Jul 08 16:57:23.426463 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:57:23.426448    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb\": container with ID starting with 33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb not found: ID does not exist" containerID="33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb"
Jul 08 16:57:23.426523 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.426470    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb"} err="failed to get container status \"33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb\": rpc error: code = NotFound desc = could not find container \"33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb\": container with ID starting with 33cc2d332480b82e5589531f70c4fb1432d0bc6add7a32133a143f296a1a41eb not found: ID does not exist"
Jul 08 16:57:23.426523 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.426488    2550 scope.go:117] "RemoveContainer" containerID="1337368cd9b095c007500288e40a38923308d71504a81eccdc589228595f6aea"
Jul 08 16:57:23.426745 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:57:23.426728    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1337368cd9b095c007500288e40a38923308d71504a81eccdc589228595f6aea\": container with ID starting with 1337368cd9b095c007500288e40a38923308d71504a81eccdc589228595f6aea not found: ID does not exist" containerID="1337368cd9b095c007500288e40a38923308d71504a81eccdc589228595f6aea"
Jul 08 16:57:23.426794 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:23.426749    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1337368cd9b095c007500288e40a38923308d71504a81eccdc589228595f6aea"} err="failed to get container status \"1337368cd9b095c007500288e40a38923308d71504a81eccdc589228595f6aea\": rpc error: code = NotFound desc = could not find container \"1337368cd9b095c007500288e40a38923308d71504a81eccdc589228595f6aea\": container with ID starting with 1337368cd9b095c007500288e40a38923308d71504a81eccdc589228595f6aea not found: ID does not exist"
Jul 08 16:57:25.027112 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.027086    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" path="/var/lib/kubelet/pods/3d6cdd1f-b5d8-4884-b60f-8e10519ee00f/volumes"
Jul 08 16:57:25.237211 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.237182    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lkfts/must-gather-5vlb2"]
Jul 08 16:57:25.237450 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.237438    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kube-rbac-proxy"
Jul 08 16:57:25.237492 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.237452    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kube-rbac-proxy"
Jul 08 16:57:25.237492 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.237464    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="storage-initializer"
Jul 08 16:57:25.237492 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.237470    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="storage-initializer"
Jul 08 16:57:25.237492 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.237483    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container"
Jul 08 16:57:25.237492 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.237489    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container"
Jul 08 16:57:25.237674 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.237537    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kube-rbac-proxy"
Jul 08 16:57:25.237674 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.237545    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d6cdd1f-b5d8-4884-b60f-8e10519ee00f" containerName="kserve-container"
Jul 08 16:57:25.241927 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.241909    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lkfts/must-gather-5vlb2"
Jul 08 16:57:25.243960 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.243941    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lkfts\"/\"openshift-service-ca.crt\""
Jul 08 16:57:25.244749 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.244734    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lkfts\"/\"default-dockercfg-fr2t4\""
Jul 08 16:57:25.244804 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.244788    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lkfts\"/\"kube-root-ca.crt\""
Jul 08 16:57:25.247854 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.247836    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lkfts/must-gather-5vlb2"]
Jul 08 16:57:25.316617 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.316590    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35715fa4-1e2e-420e-8614-834f7c2a2426-must-gather-output\") pod \"must-gather-5vlb2\" (UID: \"35715fa4-1e2e-420e-8614-834f7c2a2426\") " pod="openshift-must-gather-lkfts/must-gather-5vlb2"
Jul 08 16:57:25.316730 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.316624    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqh8x\" (UniqueName: \"kubernetes.io/projected/35715fa4-1e2e-420e-8614-834f7c2a2426-kube-api-access-cqh8x\") pod \"must-gather-5vlb2\" (UID: \"35715fa4-1e2e-420e-8614-834f7c2a2426\") " pod="openshift-must-gather-lkfts/must-gather-5vlb2"
Jul 08 16:57:25.417106 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.417060    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqh8x\" (UniqueName: \"kubernetes.io/projected/35715fa4-1e2e-420e-8614-834f7c2a2426-kube-api-access-cqh8x\") pod \"must-gather-5vlb2\" (UID: \"35715fa4-1e2e-420e-8614-834f7c2a2426\") " pod="openshift-must-gather-lkfts/must-gather-5vlb2"
Jul 08 16:57:25.417205 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.417179    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35715fa4-1e2e-420e-8614-834f7c2a2426-must-gather-output\") pod \"must-gather-5vlb2\" (UID: \"35715fa4-1e2e-420e-8614-834f7c2a2426\") " pod="openshift-must-gather-lkfts/must-gather-5vlb2"
Jul 08 16:57:25.417456 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.417439    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35715fa4-1e2e-420e-8614-834f7c2a2426-must-gather-output\") pod \"must-gather-5vlb2\" (UID: \"35715fa4-1e2e-420e-8614-834f7c2a2426\") " pod="openshift-must-gather-lkfts/must-gather-5vlb2"
Jul 08 16:57:25.424732 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.424706    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqh8x\" (UniqueName: \"kubernetes.io/projected/35715fa4-1e2e-420e-8614-834f7c2a2426-kube-api-access-cqh8x\") pod \"must-gather-5vlb2\" (UID: \"35715fa4-1e2e-420e-8614-834f7c2a2426\") " pod="openshift-must-gather-lkfts/must-gather-5vlb2"
Jul 08 16:57:25.551888 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.551859    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lkfts/must-gather-5vlb2"
Jul 08 16:57:25.672197 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.672171    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lkfts/must-gather-5vlb2"]
Jul 08 16:57:25.674355 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:57:25.674325    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35715fa4_1e2e_420e_8614_834f7c2a2426.slice/crio-4931fc52551f1abcc8846d4b53801f72523b45bd4d6b1b106927c666141a9482 WatchSource:0}: Error finding container 4931fc52551f1abcc8846d4b53801f72523b45bd4d6b1b106927c666141a9482: Status 404 returned error can't find the container with id 4931fc52551f1abcc8846d4b53801f72523b45bd4d6b1b106927c666141a9482
Jul 08 16:57:25.676129 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:25.676110    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:57:26.421974 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:26.421886    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lkfts/must-gather-5vlb2" event={"ID":"35715fa4-1e2e-420e-8614-834f7c2a2426","Type":"ContainerStarted","Data":"4931fc52551f1abcc8846d4b53801f72523b45bd4d6b1b106927c666141a9482"}
Jul 08 16:57:32.450611 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:32.450572    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lkfts/must-gather-5vlb2" event={"ID":"35715fa4-1e2e-420e-8614-834f7c2a2426","Type":"ContainerStarted","Data":"ca6db7661fdfcdbc6de7d9a2089e8c452b5eae0d37a4c4a8d41c2e3f7ff90c2f"}
Jul 08 16:57:32.450611 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:32.450618    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lkfts/must-gather-5vlb2" event={"ID":"35715fa4-1e2e-420e-8614-834f7c2a2426","Type":"ContainerStarted","Data":"bb8ba6b0b6a35c66fa50f13ee27a039230e699523358ce4cfa1783cde9b6ad74"}
Jul 08 16:57:32.467803 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:32.467755    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lkfts/must-gather-5vlb2" podStartSLOduration=1.557008867 podStartE2EDuration="7.46774034s" totalImagesPullingTime="5.910731473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:57:32.46571954 +0000 UTC m=+1368.002272016" watchObservedRunningTime="2026-07-08 16:57:32.46774034 +0000 UTC m=+1368.004292865"
Jul 08 16:57:52.538341 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:52.538311    2550 generic.go:356] "Generic (PLEG): container finished" podID="35715fa4-1e2e-420e-8614-834f7c2a2426" containerID="bb8ba6b0b6a35c66fa50f13ee27a039230e699523358ce4cfa1783cde9b6ad74" exitCode=0
Jul 08 16:57:52.538707 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:52.538378    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lkfts/must-gather-5vlb2" event={"ID":"35715fa4-1e2e-420e-8614-834f7c2a2426","Type":"ContainerDied","Data":"bb8ba6b0b6a35c66fa50f13ee27a039230e699523358ce4cfa1783cde9b6ad74"}
Jul 08 16:57:52.538707 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:52.538664    2550 scope.go:117] "RemoveContainer" containerID="bb8ba6b0b6a35c66fa50f13ee27a039230e699523358ce4cfa1783cde9b6ad74"
Jul 08 16:57:52.965596 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:52.965503    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lkfts_must-gather-5vlb2_35715fa4-1e2e-420e-8614-834f7c2a2426/gather/0.log"
Jul 08 16:57:56.078338 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:56.078292    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gm9kt_998f1936-ebcf-489b-86d0-e6ff22c6f840/global-pull-secret-syncer/0.log"
Jul 08 16:57:56.207323 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:56.207300    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-69j4d_549c7ec1-2fc1-4af7-9b87-50955730f16c/konnectivity-agent/0.log"
Jul 08 16:57:56.257096 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:56.257073    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-223.ec2.internal_f47587f1ae5d88e48c97159a168407ff/haproxy/0.log"
Jul 08 16:57:58.349228 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.349199    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-lkfts/must-gather-5vlb2"]
Jul 08 16:57:58.349545 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.349359    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-lkfts/must-gather-5vlb2" podUID="35715fa4-1e2e-420e-8614-834f7c2a2426" containerName="copy" containerID="cri-o://ca6db7661fdfcdbc6de7d9a2089e8c452b5eae0d37a4c4a8d41c2e3f7ff90c2f" gracePeriod=2
Jul 08 16:57:58.356674 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.356650    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-lkfts/must-gather-5vlb2"]
Jul 08 16:57:58.356866 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:57:58.356838    2550 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"35715fa4-1e2e-420e-8614-834f7c2a2426\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-lkfts\"/\"must-gather-5vlb2\": pods \"must-gather-5vlb2\" not found" pod="openshift-must-gather-lkfts/must-gather-5vlb2"
Jul 08 16:57:58.564097 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.564071    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lkfts_must-gather-5vlb2_35715fa4-1e2e-420e-8614-834f7c2a2426/copy/0.log"
Jul 08 16:57:58.564489 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.564464    2550 generic.go:356] "Generic (PLEG): container finished" podID="35715fa4-1e2e-420e-8614-834f7c2a2426" containerID="ca6db7661fdfcdbc6de7d9a2089e8c452b5eae0d37a4c4a8d41c2e3f7ff90c2f" exitCode=143
Jul 08 16:57:58.564583 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.564542    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4931fc52551f1abcc8846d4b53801f72523b45bd4d6b1b106927c666141a9482"
Jul 08 16:57:58.573662 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.573644    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lkfts_must-gather-5vlb2_35715fa4-1e2e-420e-8614-834f7c2a2426/copy/0.log"
Jul 08 16:57:58.575348 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.575332    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lkfts/must-gather-5vlb2"
Jul 08 16:57:58.577079 ip-10-0-138-223 kubenswrapper[2550]: E0708 16:57:58.577056    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5vlb2\" is forbidden: User \"system:node:ip-10-0-138-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lkfts\": no relationship found between node 'ip-10-0-138-223.ec2.internal' and this object" podUID="35715fa4-1e2e-420e-8614-834f7c2a2426" pod="openshift-must-gather-lkfts/must-gather-5vlb2"
Jul 08 16:57:58.677730 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.677679    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqh8x\" (UniqueName: \"kubernetes.io/projected/35715fa4-1e2e-420e-8614-834f7c2a2426-kube-api-access-cqh8x\") pod \"35715fa4-1e2e-420e-8614-834f7c2a2426\" (UID: \"35715fa4-1e2e-420e-8614-834f7c2a2426\") "
Jul 08 16:57:58.677730 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.677723    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35715fa4-1e2e-420e-8614-834f7c2a2426-must-gather-output\") pod \"35715fa4-1e2e-420e-8614-834f7c2a2426\" (UID: \"35715fa4-1e2e-420e-8614-834f7c2a2426\") "
Jul 08 16:57:58.679030 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.679011    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35715fa4-1e2e-420e-8614-834f7c2a2426-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "35715fa4-1e2e-420e-8614-834f7c2a2426" (UID: "35715fa4-1e2e-420e-8614-834f7c2a2426"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:57:58.679759 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.679741    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35715fa4-1e2e-420e-8614-834f7c2a2426-kube-api-access-cqh8x" (OuterVolumeSpecName: "kube-api-access-cqh8x") pod "35715fa4-1e2e-420e-8614-834f7c2a2426" (UID: "35715fa4-1e2e-420e-8614-834f7c2a2426"). InnerVolumeSpecName "kube-api-access-cqh8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:57:58.778481 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.778460    2550 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35715fa4-1e2e-420e-8614-834f7c2a2426-must-gather-output\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:57:58.778481 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:58.778479    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqh8x\" (UniqueName: \"kubernetes.io/projected/35715fa4-1e2e-420e-8614-834f7c2a2426-kube-api-access-cqh8x\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 08 16:57:59.027186 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:59.027136    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35715fa4-1e2e-420e-8614-834f7c2a2426" path="/var/lib/kubelet/pods/35715fa4-1e2e-420e-8614-834f7c2a2426/volumes"
Jul 08 16:57:59.570331 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:57:59.570307    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lkfts/must-gather-5vlb2"
Jul 08 16:58:00.821320 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:00.821292    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-ljv42_8edb418c-0c8e-41f4-b442-a5e6dcefb2e1/kube-state-metrics/0.log"
Jul 08 16:58:00.856063 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:00.856038    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-ljv42_8edb418c-0c8e-41f4-b442-a5e6dcefb2e1/kube-rbac-proxy-main/0.log"
Jul 08 16:58:00.886139 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:00.886117    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-ljv42_8edb418c-0c8e-41f4-b442-a5e6dcefb2e1/kube-rbac-proxy-self/0.log"
Jul 08 16:58:00.960486 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:00.960464    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-6nk8q_bca384a6-e36b-4da5-86fa-ab002dbb7ae9/monitoring-plugin/0.log"
Jul 08 16:58:01.078384 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.078334    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhf4t_8487b755-cece-4303-8108-15623916d5de/node-exporter/0.log"
Jul 08 16:58:01.105057 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.105038    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhf4t_8487b755-cece-4303-8108-15623916d5de/kube-rbac-proxy/0.log"
Jul 08 16:58:01.135508 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.135492    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhf4t_8487b755-cece-4303-8108-15623916d5de/init-textfile/0.log"
Jul 08 16:58:01.252479 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.252457    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-mmxcv_dfea924a-5429-4fbd-84f1-089e2d2a253c/kube-rbac-proxy-main/0.log"
Jul 08 16:58:01.287495 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.287470    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-mmxcv_dfea924a-5429-4fbd-84f1-089e2d2a253c/kube-rbac-proxy-self/0.log"
Jul 08 16:58:01.314628 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.314607    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-mmxcv_dfea924a-5429-4fbd-84f1-089e2d2a253c/openshift-state-metrics/0.log"
Jul 08 16:58:01.582758 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.582739    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-qdvpc_187dd64b-3c5d-4a28-bc2f-eaf1fec6deed/prometheus-operator/0.log"
Jul 08 16:58:01.614984 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.614962    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-qdvpc_187dd64b-3c5d-4a28-bc2f-eaf1fec6deed/kube-rbac-proxy/0.log"
Jul 08 16:58:01.802812 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.802792    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-96897fc96-xrbqn_6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30/thanos-query/0.log"
Jul 08 16:58:01.835819 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.835767    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-96897fc96-xrbqn_6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30/kube-rbac-proxy-web/0.log"
Jul 08 16:58:01.867352 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.867329    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-96897fc96-xrbqn_6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30/kube-rbac-proxy/0.log"
Jul 08 16:58:01.899242 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.899220    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-96897fc96-xrbqn_6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30/prom-label-proxy/0.log"
Jul 08 16:58:01.925579 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.925544    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-96897fc96-xrbqn_6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30/kube-rbac-proxy-rules/0.log"
Jul 08 16:58:01.954778 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:01.954761    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-96897fc96-xrbqn_6c3b3d2c-3fdb-4ed9-985b-142fd9fcfe30/kube-rbac-proxy-metrics/0.log"
Jul 08 16:58:03.931883 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:03.931855    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76449d99d9-sf492_335966fe-5da9-4946-85da-4040a5765c89/console/0.log"
Jul 08 16:58:04.243542 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.243464    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"]
Jul 08 16:58:04.243846 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.243828    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35715fa4-1e2e-420e-8614-834f7c2a2426" containerName="gather"
Jul 08 16:58:04.243930 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.243849    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="35715fa4-1e2e-420e-8614-834f7c2a2426" containerName="gather"
Jul 08 16:58:04.243930 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.243866    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35715fa4-1e2e-420e-8614-834f7c2a2426" containerName="copy"
Jul 08 16:58:04.243930 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.243874    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="35715fa4-1e2e-420e-8614-834f7c2a2426" containerName="copy"
Jul 08 16:58:04.244082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.243947    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="35715fa4-1e2e-420e-8614-834f7c2a2426" containerName="copy"
Jul 08 16:58:04.244082 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.243962    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="35715fa4-1e2e-420e-8614-834f7c2a2426" containerName="gather"
Jul 08 16:58:04.247208 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.247189    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.249508 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.249485    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hpwtp\"/\"kube-root-ca.crt\""
Jul 08 16:58:04.249649 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.249510    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hpwtp\"/\"default-dockercfg-n7g2q\""
Jul 08 16:58:04.250671 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.250498    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hpwtp\"/\"openshift-service-ca.crt\""
Jul 08 16:58:04.254113 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.254090    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"]
Jul 08 16:58:04.315802 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.315777    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49e03ed3-0a59-4528-a65d-92895ebf49bf-podres\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.315905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.315843    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49e03ed3-0a59-4528-a65d-92895ebf49bf-proc\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.315905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.315863    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8kp5\" (UniqueName: \"kubernetes.io/projected/49e03ed3-0a59-4528-a65d-92895ebf49bf-kube-api-access-w8kp5\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.315905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.315885    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49e03ed3-0a59-4528-a65d-92895ebf49bf-sys\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.315905 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.315901    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49e03ed3-0a59-4528-a65d-92895ebf49bf-lib-modules\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.416602 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.416533    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49e03ed3-0a59-4528-a65d-92895ebf49bf-proc\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.416800 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.416615    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8kp5\" (UniqueName: \"kubernetes.io/projected/49e03ed3-0a59-4528-a65d-92895ebf49bf-kube-api-access-w8kp5\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.416800 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.416647    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49e03ed3-0a59-4528-a65d-92895ebf49bf-proc\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.416800 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.416657    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49e03ed3-0a59-4528-a65d-92895ebf49bf-sys\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.416800 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.416706    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49e03ed3-0a59-4528-a65d-92895ebf49bf-lib-modules\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.416800 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.416727    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49e03ed3-0a59-4528-a65d-92895ebf49bf-sys\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.416800 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.416731    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49e03ed3-0a59-4528-a65d-92895ebf49bf-podres\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.417023 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.416838    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49e03ed3-0a59-4528-a65d-92895ebf49bf-podres\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.417023 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.416877    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49e03ed3-0a59-4528-a65d-92895ebf49bf-lib-modules\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.425162 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.425137    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8kp5\" (UniqueName: \"kubernetes.io/projected/49e03ed3-0a59-4528-a65d-92895ebf49bf-kube-api-access-w8kp5\") pod \"perf-node-gather-daemonset-9n8xr\" (UID: \"49e03ed3-0a59-4528-a65d-92895ebf49bf\") " pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.556764 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.556742    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:04.681315 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:04.681290    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"]
Jul 08 16:58:04.682610 ip-10-0-138-223 kubenswrapper[2550]: W0708 16:58:04.682586    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod49e03ed3_0a59_4528_a65d_92895ebf49bf.slice/crio-4f0dbaa3e0d18778b0cdd8f4df3d8cbf1c0131c82a2a6f083860334e7a6a45b4 WatchSource:0}: Error finding container 4f0dbaa3e0d18778b0cdd8f4df3d8cbf1c0131c82a2a6f083860334e7a6a45b4: Status 404 returned error can't find the container with id 4f0dbaa3e0d18778b0cdd8f4df3d8cbf1c0131c82a2a6f083860334e7a6a45b4
Jul 08 16:58:05.594499 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:05.594463    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr" event={"ID":"49e03ed3-0a59-4528-a65d-92895ebf49bf","Type":"ContainerStarted","Data":"dc0166ab9be0b5383bcb98fb7f98d9ec00d22f79cd208e30ccfba680e0acf826"}
Jul 08 16:58:05.594889 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:05.594546    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr" event={"ID":"49e03ed3-0a59-4528-a65d-92895ebf49bf","Type":"ContainerStarted","Data":"4f0dbaa3e0d18778b0cdd8f4df3d8cbf1c0131c82a2a6f083860334e7a6a45b4"}
Jul 08 16:58:05.594889 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:05.594688    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:05.616998 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:05.616960    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr" podStartSLOduration=1.616948874 podStartE2EDuration="1.616948874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:58:05.616169371 +0000 UTC m=+1401.152721857" watchObservedRunningTime="2026-07-08 16:58:05.616948874 +0000 UTC m=+1401.153501395"
Jul 08 16:58:05.651435 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:05.651415    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rnngg_029df44a-7597-4aaa-820e-16e4e645f062/dns/0.log"
Jul 08 16:58:05.687156 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:05.687139    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rnngg_029df44a-7597-4aaa-820e-16e4e645f062/kube-rbac-proxy/0.log"
Jul 08 16:58:05.724827 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:05.724803    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4dpm5_248e79a6-fa20-4328-a4e7-0c6c54c93cd9/dns-node-resolver/0.log"
Jul 08 16:58:06.332948 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:06.332932    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kxzc4_1dcc1bed-da11-489e-801a-904a8a21583a/node-ca/0.log"
Jul 08 16:58:07.487885 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:07.487859    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j9wp8_277ceaee-f329-4dde-b0a0-d46493d0acaa/serve-healthcheck-canary/0.log"
Jul 08 16:58:07.932580 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:07.932532    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gk7dn_686acd8e-2b2b-441f-8502-cce0d131d3f3/kube-rbac-proxy/0.log"
Jul 08 16:58:07.955605 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:07.955588    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gk7dn_686acd8e-2b2b-441f-8502-cce0d131d3f3/exporter/0.log"
Jul 08 16:58:07.978962 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:07.978947    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gk7dn_686acd8e-2b2b-441f-8502-cce0d131d3f3/extractor/0.log"
Jul 08 16:58:10.424867 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:10.424838    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-x6dqm_22b7e784-74ba-491a-841a-2f252eb682c1/s3-init/0.log"
Jul 08 16:58:10.442049 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:10.442024    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-x6dqm_22b7e784-74ba-491a-841a-2f252eb682c1/download-hf-model/0.log"
Jul 08 16:58:10.469938 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:10.469901    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-qxlf9_b69b200d-32dc-4711-9127-b7a62b85a04e/seaweedfs/0.log"
Jul 08 16:58:11.606958 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:11.606921    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hpwtp/perf-node-gather-daemonset-9n8xr"
Jul 08 16:58:16.022528 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:16.022493    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp7pp_d2c866b7-440e-4458-bfdf-fc786d235674/kube-multus-additional-cni-plugins/0.log"
Jul 08 16:58:16.043656 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:16.043633    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp7pp_d2c866b7-440e-4458-bfdf-fc786d235674/egress-router-binary-copy/0.log"
Jul 08 16:58:16.063730 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:16.063713    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp7pp_d2c866b7-440e-4458-bfdf-fc786d235674/cni-plugins/0.log"
Jul 08 16:58:16.084634 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:16.084613    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp7pp_d2c866b7-440e-4458-bfdf-fc786d235674/bond-cni-plugin/0.log"
Jul 08 16:58:16.105289 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:16.105265    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp7pp_d2c866b7-440e-4458-bfdf-fc786d235674/routeoverride-cni/0.log"
Jul 08 16:58:16.127173 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:16.127157    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp7pp_d2c866b7-440e-4458-bfdf-fc786d235674/whereabouts-cni-bincopy/0.log"
Jul 08 16:58:16.147155 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:16.147123    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp7pp_d2c866b7-440e-4458-bfdf-fc786d235674/whereabouts-cni/0.log"
Jul 08 16:58:16.368355 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:16.368338    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-x8vzr_b7106a7f-6389-4c33-ac08-b08cde3dc850/kube-multus/0.log"
Jul 08 16:58:16.485409 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:16.485367    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xphrq_787b3d58-fe95-4bac-a035-bd65399a7f30/network-metrics-daemon/0.log"
Jul 08 16:58:16.508387 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:16.508373    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xphrq_787b3d58-fe95-4bac-a035-bd65399a7f30/kube-rbac-proxy/0.log"
Jul 08 16:58:17.689711 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:17.689686    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzzb2_2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3/ovn-controller/0.log"
Jul 08 16:58:17.726174 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:17.726151    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzzb2_2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3/ovn-acl-logging/0.log"
Jul 08 16:58:17.750064 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:17.750040    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzzb2_2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3/kube-rbac-proxy-node/0.log"
Jul 08 16:58:17.783255 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:17.783235    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzzb2_2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3/kube-rbac-proxy-ovn-metrics/0.log"
Jul 08 16:58:17.813077 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:17.813060    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzzb2_2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3/northd/0.log"
Jul 08 16:58:17.839237 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:17.839217    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzzb2_2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3/nbdb/0.log"
Jul 08 16:58:17.870106 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:17.870075    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzzb2_2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3/sbdb/0.log"
Jul 08 16:58:17.911121 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:17.911099    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzzb2_2d2ae27f-9ece-4e08-a9df-aa3c0153b3d3/ovnkube-controller/0.log"
Jul 08 16:58:18.889433 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:18.889408    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-79p9f_d22dbf53-7ebd-4312-a491-894b8392147d/network-check-target-container/0.log"
Jul 08 16:58:19.867391 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:19.867366    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-t6n9h_bb3cafd2-b2e5-44b5-8f40-36e9e70b9551/iptables-alerter/0.log"
Jul 08 16:58:20.476634 ip-10-0-138-223 kubenswrapper[2550]: I0708 16:58:20.476614    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-2sszl_816f0dd7-4e9f-49d8-a0d3-fb152a6e3146/tuned/0.log"