Jul 09 18:02:09.421363 ip-10-0-130-107 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 09 18:02:09.421375 ip-10-0-130-107 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 09 18:02:09.421384 ip-10-0-130-107 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 09 18:02:09.421698 ip-10-0-130-107 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 09 18:02:20.657647 ip-10-0-130-107 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 09 18:02:20.657666 ip-10-0-130-107 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 25e952e66cec4e8cb93251c4c1d2a831 --
Jul 09 18:04:23.039830 ip-10-0-130-107 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.516584    2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520896    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520909    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520914    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520917    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520921    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520924    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520927    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:23.574787 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520930    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520932    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520935    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520938    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520940    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520943    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520945    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520948    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520950    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520953    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520957    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520959    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520971    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520976    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520979    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520983    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520986    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520989    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520991    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520994    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:23.576486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.520997    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521000    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521002    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521005    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521008    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521010    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521013    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521015    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521018    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521021    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521023    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521026    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521030    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521034    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521037    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521040    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521043    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521045    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521048    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:23.577130 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521050    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521053    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521055    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521058    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521061    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521063    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521068    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521071    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521076    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521079    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521083    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521086    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521089    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521092    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521095    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521097    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521100    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521103    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521106    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521109    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:23.578432 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521112    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521115    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521118    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521121    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521124    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521127    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521130    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521132    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521135    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521138    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521140    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521143    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521146    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521148    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521152    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521154    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521157    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521160    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521162    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521165    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:23.579091 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521168    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521170    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521173    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521175    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521178    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521181    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521184    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521186    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521189    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521191    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521194    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521196    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521199    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521888    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521894    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521897    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521900    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521903    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521911    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521914    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:23.580771 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521916    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521919    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521922    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521925    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521927    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521930    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521932    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521938    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521942    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521945    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521949    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521952    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521954    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521957    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521960    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521963    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521966    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521968    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521971    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:23.581872 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521974    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521977    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521979    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521982    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521985    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521987    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521990    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521993    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521996    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.521998    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522001    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522003    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522006    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522009    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522011    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522014    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522017    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522021    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522024    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522027    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:23.584929 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522030    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522033    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522036    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522038    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522041    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522044    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522046    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522049    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522052    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522055    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522057    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522060    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522063    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522066    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522068    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522071    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522073    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522076    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522078    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522082    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:23.585986 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522084    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522087    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522090    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522092    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522105    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522108    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522111    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522114    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522117    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522119    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522122    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522124    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522127    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522130    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522133    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522135    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522138    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522142    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522146    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522149    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:23.588352 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522152    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522155    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522158    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522161    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522164    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522167    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522169    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522172    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522175    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522177    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522180    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522182    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522185    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522581    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522588    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522591    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522594    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522597    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522600    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522603    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:23.588964 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522608    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522611    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522615    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522618    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522620    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522623    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522626    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522628    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522631    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522634    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522637    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522639    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522642    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522645    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522647    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522652    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522655    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522658    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522661    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522664    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:23.592163 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522666    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522669    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522671    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522674    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522677    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522680    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522683    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522685    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522688    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522691    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522694    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522696    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522699    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522702    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522704    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522707    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522709    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522711    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522714    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522717    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:23.592814 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522719    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522721    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522724    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522726    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522728    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522731    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522734    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522736    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522739    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522741    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522744    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522747    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522749    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522752    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522754    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522757    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522760    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522762    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522765    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:23.596549 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522767    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:23.594915 ip-10-0-130-107 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522770    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522775    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522778    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522781    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522783    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522786    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522788    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522791    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522793    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522796    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522798    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522801    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522803    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522806    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522809    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522811    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522815    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522818    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522823    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:23.598083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522825    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522828    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522830    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522833    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522837    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522840    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522843    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522846    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522849    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522852    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522855    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522858    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.522861    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522938    2572 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522949    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522958    2572 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522962    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522967    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522970    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522974    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:04:23.600636 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522978    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522981    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522984    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522988    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522991    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522994    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.522997    2572 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523000    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523003    2572 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523006    2572 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523009    2572 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523014    2572 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523017    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523020    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523023    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523026    2572 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523030    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523033    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523036    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523039    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523042    2572 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523045    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523048    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523051    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:04:23.601734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523054    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523058    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523063    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523065    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523068    2572 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523071    2572 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523075    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523079    2572 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523082    2572 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523085    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523088    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523091    2572 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523095    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523098    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523101    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523104    2572 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523107    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523110    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523112    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523130    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523134    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523138    2572 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523141    2572 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523145    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523148    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:04:23.604125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523151    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523154    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523157    2572 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523160    2572 flags.go:64] FLAG: --help="false"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523163    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-130-107.ec2.internal"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523166    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523169    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523172    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523176    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523182    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523185    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523188    2572 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523191    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523194    2572 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523201    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523204    2572 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523207    2572 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523210    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523213    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523216    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523219    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523222    2572 flags.go:64] FLAG: --lock-file=""
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523224    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523227    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:04:23.605424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523230    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523235    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523240    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523243    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523245    2572 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523248    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523251    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523254    2572 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523257    2572 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523262    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523265    2572 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523269    2572 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523272    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523275    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523278    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523281    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523283    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523288    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523291    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523298    2572 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523301    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523304    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523307    2572 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:04:23.608514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523311    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523316    2572 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523319    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523322    2572 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523325    2572 flags.go:64] FLAG: --port="10250"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523328    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523331    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05b4d1b7bba9d383f"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523334    2572 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523338    2572 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523341    2572 flags.go:64] FLAG: --register-node="true"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523344    2572 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523347    2572 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523351    2572 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523354    2572 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523357    2572 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523361    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523364    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523367    2572 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523370    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523373    2572 flags.go:64] FLAG: --runonce="false"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523375    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523378    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523381    2572 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523384    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523387    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523390    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:04:23.609167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523395    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523398    2572 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523401    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523403    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523406    2572 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523409    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523412    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523415    2572 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523418    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523423    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523426    2572 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523429    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523433    2572 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523436    2572 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523438    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523441    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523444    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523447    2572 flags.go:64] FLAG: --v="2"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523452    2572 flags.go:64] FLAG: --version="false"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523456    2572 flags.go:64] FLAG: --vmodule=""
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523459    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523462    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523602    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523608    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:23.680063 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523611    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523614    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523618    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523621    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523624    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523627    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523629    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523633    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523637    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523640    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523642    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523645    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523648    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523651    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523654    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523657    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523665    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523668    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523673    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:23.681539 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523682    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523686    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523689    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523691    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523694    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523697    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523699    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523704    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523706    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523709    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523712    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523714    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523717    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523720    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523724    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523726    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523729    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523732    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523734    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523737    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:23.683955 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523739    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523744    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523747    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523751    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523754    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523757    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523759    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523762    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523765    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523768    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523776    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523779    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523782    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523784    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523787    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523790    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523793    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523795    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523797    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:23.685008 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523801    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523804    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523807    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523809    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523812    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523814    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523817    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523820    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523822    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523825    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523827    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523830    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523833    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523835    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523839    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523842    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523844    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523847    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523850    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523852    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:23.687302 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523855    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523857    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523860    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523862    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523870    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523872    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523875    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523877    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523880    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523882    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523885    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523887    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523891    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523894    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523896    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523899    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523901    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523904    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:23.688000 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.523907    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.523912    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.530843    2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.530861    2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.531662    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.531672    2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531748    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531756    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531761    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531766    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531771    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531775    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531779    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531783    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531787    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:23.691083 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531790    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531794    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531799    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531804    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531808    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531812    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531815    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531820    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531823    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531827    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531831    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531835    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531839    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531843    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531846    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531850    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531854    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531860    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531865    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531870    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:23.691998 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531875    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531879    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531883    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531888    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531891    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531894    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531897    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531899    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531903    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531907    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531910    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531915    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531918    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531922    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531926    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531930    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531935    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531940    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531943    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531947    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:23.696533 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531950    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531954    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531958    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531962    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531965    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531970    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531976    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531980    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531984    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531988    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531992    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531996    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.531999    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532004    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532008    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532011    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532015    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532019    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532022    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:23.697471 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532026    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532030    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532034    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532037    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532041    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532047    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532051    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532055    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532059    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532062    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532067    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532071    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532075    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532079    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532082    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532086    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532090    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532093    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532099    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:23.698184 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532106    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532111    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532115    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532118    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532121    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532124    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532127    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532130    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532133    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532136    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532138    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532141    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.532146    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532262    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:23.701425 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532267    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532269    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532272    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532275    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532279    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532282    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532284    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532286    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532290    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532294    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532298    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532300    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532303    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532305    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532308    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532311    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532315    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532318    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532321    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:23.702132 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532326    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532329    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532332    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532335    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532338    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532340    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532343    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532346    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532348    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532351    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532353    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532356    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532358    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532361    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532363    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532366    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532368    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532372    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532375    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532377    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:23.702831 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532380    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532383    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532385    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532388    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532391    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532393    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532396    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532399    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532401    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532404    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532406    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532408    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532413    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532415    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532418    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532421    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532423    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532426    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532428    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532431    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:23.704909 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532433    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532436    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532439    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532441    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532444    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532446    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532449    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532451    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532454    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532458    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532461    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532465    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532468    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532471    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532474    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532476    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532479    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532482    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532485    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:23.705594 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532488    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532490    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532492    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532495    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532497    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532501    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532504    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532506    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532509    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532511    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532514    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532517    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532519    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532521    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532524    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532526    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532529    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532532    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532534    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:23.792382 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:23.532537    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:23.811159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.532542    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:23.811159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.532652    2572 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:04:23.811159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.534625    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:04:23.811159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.536008    2572 server.go:1019] "Starting client certificate rotation"
Jul 09 18:04:23.811159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.536107    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:23.811159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.536173    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:23.811159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.547646    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:23.811159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.548773    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:23.811159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.558220    2572 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:04:23.811159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.560368    2572 log.go:25] "Validated CRI v1 image API"
Jul 09 18:04:23.811159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.561744    2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:04:23.811159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.565733    2572 fs.go:135] Filesystem UUIDs: map[2db3eb47-ce66-423b-bab1-96192895f9fc:/dev/nvme0n1p4 4e992b67-3a0d-41bd-ac9f-d1ed36e668e5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 09 18:04:23.813930 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.565748    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:04:23.814156 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.569873    2572 manager.go:216] Machine: {Timestamp:2026-07-09 18:04:23.56939618 +0000 UTC m=+0.407739155 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100780 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f7e99700defd1cadd55412b893f50 SystemUUID:ec2f7e99-700d-efd1-cadd-55412b893f50 BootID:25e952e6-6cec-4e8c-b932-51c4c1d2a831 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e3:bb:04:72:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e3:bb:04:72:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:cd:20:da:b5:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:04:23.814156 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.569965    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:04:23.814156 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.570035    2572 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:04:23.814156 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.573428    2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:04:23.814156 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.573454    2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-107.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:04:23.814156 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.574694    2572 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:04:23.814156 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.574706    2572 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:04:23.814156 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.574713    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.575545    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.576847    2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.579824    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.579945    2572 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.580721    2572 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.580736    2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.580751    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.580760    2572 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.580770    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.581136    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.581157    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.583728    2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.587759    2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.587803    2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.588002    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.588019    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.588033    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.588052    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.588064    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.588072    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.589833    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.589848    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.589859    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.589866    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.589877    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.589887    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.590647    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.590856    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.594060    2572 server.go:1268] "Started kubelet"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.594165    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.594204    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.594263    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.594967    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.594983    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.594983    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.595153    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-107.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.595392    2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:04:23.901232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.596196    2572 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.599411    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-107.ec2.internal.18c0b12e01b5fdae  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-107.ec2.internal,UID:ip-10-0-130-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-107.ec2.internal,},FirstTimestamp:2026-07-09 18:04:23.594032558 +0000 UTC m=+0.432375533,LastTimestamp:2026-07-09 18:04:23.594032558 +0000 UTC m=+0.432375533,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-107.ec2.internal,}"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.600849    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.600851    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.601011    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.601471    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.601512    2572 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.601530    2572 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.601629    2572 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.601639    2572 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.620240    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.620477    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.629026    2572 factory.go:55] Registering systemd factory
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.629039    2572 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.629460    2572 factory.go:153] Registering CRI-O factory
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.629471    2572 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.629512    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.629535    2572 factory.go:103] Registering Raw factory
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.629546    2572 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.629786    2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.629991    2572 manager.go:349] Starting recovery of all containers
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.640956    2572 manager.go:354] Recovery completed
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.646188    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.648174    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.648211    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.648222    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.648685    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.648695    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.648711    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.648746    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.649420    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.649460    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.649475    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:23.903016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.649972    2572 policy_none.go:49] "None policy: Start"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.649986    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.649997    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.651186    2572 policy_none.go:47] "Start"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.654612    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z4vh4"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.660699    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-107.ec2.internal.18c0b12e04f07795  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-107.ec2.internal,UID:ip-10-0-130-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-107.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-107.ec2.internal,},FirstTimestamp:2026-07-09 18:04:23.648196501 +0000 UTC m=+0.486539479,LastTimestamp:2026-07-09 18:04:23.648196501 +0000 UTC m=+0.486539479,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-107.ec2.internal,}"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.664800    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z4vh4"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.678069    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-107.ec2.internal.18c0b12e04f0c528  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-107.ec2.internal,UID:ip-10-0-130-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-107.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-107.ec2.internal,},FirstTimestamp:2026-07-09 18:04:23.64821636 +0000 UTC m=+0.486559338,LastTimestamp:2026-07-09 18:04:23.64821636 +0000 UTC m=+0.486559338,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-107.ec2.internal,}"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.701700    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.738304    2572 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.738343    2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.738356    2572 server.go:87] "Starting device plugin registration server"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.738609    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.738620    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.738900    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.738976    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.738984    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.739544    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.739608    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-107.ec2.internal\" not found"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.828818    2572 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-107.ec2.internal\" not found" node="ip-10-0-130-107.ec2.internal"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.839321    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.840274    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.840308    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.840321    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.840348    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-107.ec2.internal"
Jul 09 18:04:23.904158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.849290    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-107.ec2.internal"
Jul 09 18:04:23.905051 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.849310    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-107.ec2.internal\": node \"ip-10-0-130-107.ec2.internal\" not found"
Jul 09 18:04:23.905051 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.879048    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found"
Jul 09 18:04:23.926054 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.926031    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:04:23.927388 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.927365    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:04:23.927499 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.927397    2572 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:04:23.927499 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.927419    2572 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:04:23.927499 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.927452    2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:04:23.931777 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:23.931755    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:23.980049 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:23.980011    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found"
Jul 09 18:04:24.029072 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.029046    2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal"]
Jul 09 18:04:24.029162 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.029119    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:24.030772 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.030756    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:24.030875 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.030794    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:24.030875 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.030809    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:24.032151 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.032135    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:24.032246 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.032235    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.032299 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.032259    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:24.032910 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.032892    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:24.032967 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.032922    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:24.032967 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.032932    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:24.033029 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.032896    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:24.033029 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.033001    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:24.033029 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.033011    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:24.034227 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.034209    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.034304 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.034241    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:24.034899 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.034885    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:24.034973 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.034911    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:24.034973 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.034924    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:24.054192 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.054168    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-107.ec2.internal\" not found" node="ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.058667 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.058650    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-107.ec2.internal\" not found" node="ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.080633 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.080612    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found"
Jul 09 18:04:24.104859 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.104839    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09ef526d198a51a2dc5e772145e24829-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal\" (UID: \"09ef526d198a51a2dc5e772145e24829\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.104909 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.104864    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c54f981040f96dfb2de508fe0606b9e4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-107.ec2.internal\" (UID: \"c54f981040f96dfb2de508fe0606b9e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.104909 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.104881    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09ef526d198a51a2dc5e772145e24829-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal\" (UID: \"09ef526d198a51a2dc5e772145e24829\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.181685 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.181619    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found"
Jul 09 18:04:24.205739 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.205709    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c54f981040f96dfb2de508fe0606b9e4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-107.ec2.internal\" (UID: \"c54f981040f96dfb2de508fe0606b9e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.205822 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.205747    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09ef526d198a51a2dc5e772145e24829-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal\" (UID: \"09ef526d198a51a2dc5e772145e24829\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.205822 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.205764    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09ef526d198a51a2dc5e772145e24829-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal\" (UID: \"09ef526d198a51a2dc5e772145e24829\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.205822 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.205810    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09ef526d198a51a2dc5e772145e24829-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal\" (UID: \"09ef526d198a51a2dc5e772145e24829\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.205822 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.205815    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c54f981040f96dfb2de508fe0606b9e4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-107.ec2.internal\" (UID: \"c54f981040f96dfb2de508fe0606b9e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.205993 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.205815    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09ef526d198a51a2dc5e772145e24829-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal\" (UID: \"09ef526d198a51a2dc5e772145e24829\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.281885 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.281853    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found"
Jul 09 18:04:24.360265 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.360233    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.361822 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.361808    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.382082 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.382048    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-107.ec2.internal\" not found"
Jul 09 18:04:24.446545 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.446464    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:24.501526 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.501494    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.520409 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.520388    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:24.522613 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.522599    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.534893 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.534873    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:04:24.534999 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.534981    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:24.535042 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.534987    2572 request.go:1196] "Unexpected error when reading response body" err="read tcp 10.0.130.107:33436->100.57.176.33:6443: use of closed network connection"
Jul 09 18:04:24.535042 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.535001    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:24.535042 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.535026    2572 kubelet.go:3250] "Failed creating a mirror pod" err="unexpected error when reading response body. Please retry. Original error: read tcp 10.0.130.107:33436->100.57.176.33:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal"
Jul 09 18:04:24.550831 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.550802    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:24.581588 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.581550    2572 apiserver.go:52] "Watching apiserver"
Jul 09 18:04:24.591146 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.591125    2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:04:24.592775 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.592746    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-c9jnd","kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n","openshift-cluster-node-tuning-operator/tuned-hg7xk","openshift-multus/multus-additional-cni-plugins-t9r9n","openshift-network-operator/iptables-alerter-bnnxp","openshift-ovn-kubernetes/ovnkube-node-hmxsn","kube-system/global-pull-secret-syncer-j6pks","kube-system/konnectivity-agent-srlzs","openshift-dns/node-resolver-tgwm7","openshift-image-registry/node-ca-6fbtz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal","openshift-multus/multus-78dlw","openshift-multus/network-metrics-daemon-pw25w"]
Jul 09 18:04:24.595655 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.595630    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.597020 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.596996    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.598196 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.598176    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rhh5w\""
Jul 09 18:04:24.598278 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.598236    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.598805 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.598788    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:04:24.599146 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.599125    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:04:24.599283 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.599174    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:04:24.599283 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.599174    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:04:24.599406 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.599297    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:04:24.599455 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.599440    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.599523 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.599508    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:04:24.600885 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.600867    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.600977 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.600951    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:24.601235 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.601211    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:04:24.601327 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.601238    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:04:24.601327 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.601264    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:04:24.601327 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.601293    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:04:24.601474 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.601327    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:04:24.601474 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.601406    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:04:24.601615 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.601557    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:04:24.601791 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.601776    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-67zj2\""
Jul 09 18:04:24.602218 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.602193    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h7sj8\""
Jul 09 18:04:24.602218 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.602213    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:24.602387 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.602292    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:24.602442 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.602402    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bnnxp"
Jul 09 18:04:24.603329 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.603307    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2mpbg\""
Jul 09 18:04:24.603955 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.603939    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-srlzs"
Jul 09 18:04:24.604050 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.604016    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dm6h4\""
Jul 09 18:04:24.604131 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.604112    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:04:24.604622 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.604604    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:04:24.605249 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.605236    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:24.605318 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.605300    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:24.606558 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.606516    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:24.606742 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.606599    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:24.606742 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.606703    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:24.606886 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.606810    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:24.606886 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.606845    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rp9v2\""
Jul 09 18:04:24.607224 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607204    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-kubelet\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.607302 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607228    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-var-lib-cni-bin\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.607302 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607245    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.607302 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607266    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-sys-fs\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.607447 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607318    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-host\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.607447 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607337    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a690986-5200-4f8f-803d-1db7de256ddb-cnibin\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.607447 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607353    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8a690986-5200-4f8f-803d-1db7de256ddb-os-release\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.607447 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607392    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-hostroot\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.607447 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607421    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-sysconfig\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.607447 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607438    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a690986-5200-4f8f-803d-1db7de256ddb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.607745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607456    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-var-lib-openvswitch\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.607745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607491    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-run-openvswitch\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.607745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-kubernetes\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.607745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607540    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-var-lib-kubelet\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.607745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607581    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0281d37-6c54-407c-bc65-61d01eeb63b9-iptables-alerter-script\") pod \"iptables-alerter-bnnxp\" (UID: \"a0281d37-6c54-407c-bc65-61d01eeb63b9\") " pod="openshift-network-operator/iptables-alerter-bnnxp"
Jul 09 18:04:24.607745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607630    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-slash\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.607745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607654    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-os-release\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.607745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607678    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-multus-socket-dir-parent\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.607745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607701    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-var-lib-cni-multus\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.607745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607718    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-sys\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.607745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607733    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a690986-5200-4f8f-803d-1db7de256ddb-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.607745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607748    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-cni-bin\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607764    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-device-dir\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607791    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607808    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6c1dc49-8503-40c7-b286-0d1b9706aec6-ovnkube-config\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607833    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6c1dc49-8503-40c7-b286-0d1b9706aec6-env-overrides\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607873    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e716b8d-7461-4f3f-b8e2-63a243537324-cni-binary-copy\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607896    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-run-multus-certs\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607922    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-etc-selinux\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607943    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-sysctl-d\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607959    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-lib-modules\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.607977    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-tuned\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608001    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-run-netns\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608024    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvklb\" (UniqueName: \"kubernetes.io/projected/1e716b8d-7461-4f3f-b8e2-63a243537324-kube-api-access-cvklb\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.608318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608052    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f542j\" (UniqueName: \"kubernetes.io/projected/a0281d37-6c54-407c-bc65-61d01eeb63b9-kube-api-access-f542j\") pod \"iptables-alerter-bnnxp\" (UID: \"a0281d37-6c54-407c-bc65-61d01eeb63b9\") " pod="openshift-network-operator/iptables-alerter-bnnxp"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608103    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a690986-5200-4f8f-803d-1db7de256ddb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608129    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-etc-openvswitch\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608154    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-run-k8s-cni-cncf-io\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608180    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-registration-dir\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608202    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-tmp\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608226    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a690986-5200-4f8f-803d-1db7de256ddb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608250    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-system-cni-dir\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608278    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e716b8d-7461-4f3f-b8e2-63a243537324-multus-daemon-config\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608290    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tgwm7"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608302    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-cni-netd\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608339    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6c1dc49-8503-40c7-b286-0d1b9706aec6-ovnkube-script-lib\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608358    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmg9x\" (UniqueName: \"kubernetes.io/projected/423e031a-a902-482b-8cf0-050a08d3260d-kube-api-access-fmg9x\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608387    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-systemd\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.608992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608413    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-run\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608442    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9vxc\" (UniqueName: \"kubernetes.io/projected/8a690986-5200-4f8f-803d-1db7de256ddb-kube-api-access-s9vxc\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608466    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0281d37-6c54-407c-bc65-61d01eeb63b9-host-slash\") pod \"iptables-alerter-bnnxp\" (UID: \"a0281d37-6c54-407c-bc65-61d01eeb63b9\") " pod="openshift-network-operator/iptables-alerter-bnnxp"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608492    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-run-ovn\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608514    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-multus-cni-dir\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608534    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-etc-kubernetes\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608565    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-sysctl-conf\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608599    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-cnibin\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608625    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-systemd-units\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608644    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-node-log\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608686    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-log-socket\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608702    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-var-lib-kubelet\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608719    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxrtp\" (UniqueName: \"kubernetes.io/projected/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-kube-api-access-mxrtp\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608734    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjgp\" (UniqueName: \"kubernetes.io/projected/a6c1dc49-8503-40c7-b286-0d1b9706aec6-kube-api-access-lhjgp\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.609705 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608750    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-run-netns\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.610257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608798    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-socket-dir\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.610257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608827    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-modprobe-d\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.610257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608844    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:04:24.610257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608854    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6c1dc49-8503-40c7-b286-0d1b9706aec6-ovn-node-metrics-cert\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.610257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608880    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:04:24.610257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608886    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-multus-conf-dir\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.610257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608955    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a690986-5200-4f8f-803d-1db7de256ddb-system-cni-dir\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.610257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.608983    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-run-systemd\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.610257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.609006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-run-ovn-kubernetes\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.610257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.609107    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-648ww\""
Jul 09 18:04:24.610257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.609578    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6fbtz"
Jul 09 18:04:24.610257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.609874    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:04:24.611021 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.611002    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:24.611084 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.611065    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:24.611220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.611205    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:04:24.611384 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.611311    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vc79s\""
Jul 09 18:04:24.611534 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.611510    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:04:24.612393 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.612378    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:04:24.612814 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.612801    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:04:24.613244 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.613231    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xfc58\""
Jul 09 18:04:24.613464 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.613449    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:24.613721 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.613707    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:04:24.651283 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.651266    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7hg4p"
Jul 09 18:04:24.660616 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.660596    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7hg4p"
Jul 09 18:04:24.674186 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.674137    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:59:24 +0000 UTC" deadline="2028-03-10 04:26:54.64711196 +0000 UTC"
Jul 09 18:04:24.674186 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.674181    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14626h22m29.972933363s"
Jul 09 18:04:24.702462 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.702439    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:04:24.709187 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709166    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-var-lib-cni-bin\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.709263 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709197    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.709263 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709217    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-sys-fs\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.709263 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709240    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-host\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.709368 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709267    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a690986-5200-4f8f-803d-1db7de256ddb-cnibin\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.709368 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709284    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-var-lib-cni-bin\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.709368 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709293    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8a690986-5200-4f8f-803d-1db7de256ddb-os-release\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.709368 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709322    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-hostroot\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.709368 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709349    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-sysconfig\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709368    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8a690986-5200-4f8f-803d-1db7de256ddb-os-release\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709377    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a690986-5200-4f8f-803d-1db7de256ddb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709362    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-host\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709320    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a690986-5200-4f8f-803d-1db7de256ddb-cnibin\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709412    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-var-lib-openvswitch\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709420    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-hostroot\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709457    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-sys-fs\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709493    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-sysconfig\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709514    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709532    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-var-lib-openvswitch\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709569    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709576    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-run-openvswitch\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.709599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709600    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-dbus\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709616    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf5jk\" (UniqueName: \"kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk\") pod \"network-check-target-c9jnd\" (UID: \"d50047de-b0e9-4a3c-99ad-b4566e13172c\") " pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709633    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-kubernetes\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709649    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-var-lib-kubelet\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709667    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0281d37-6c54-407c-bc65-61d01eeb63b9-iptables-alerter-script\") pod \"iptables-alerter-bnnxp\" (UID: \"a0281d37-6c54-407c-bc65-61d01eeb63b9\") " pod="openshift-network-operator/iptables-alerter-bnnxp"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709693    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-slash\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709721    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-os-release\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709689    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-run-openvswitch\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709725    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-kubernetes\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709738    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-var-lib-kubelet\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709758    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-multus-socket-dir-parent\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709775    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-slash\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709794    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-os-release\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709793    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-var-lib-cni-multus\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709821    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-sys\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.710251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709843    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a690986-5200-4f8f-803d-1db7de256ddb-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-cni-bin\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709866    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-var-lib-cni-multus\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709890    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-device-dir\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709903    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-sys\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709910    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-multus-socket-dir-parent\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709917    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709942    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-cni-bin\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709967    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-device-dir\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709970    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.709997    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6c1dc49-8503-40c7-b286-0d1b9706aec6-ovnkube-config\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710021    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6c1dc49-8503-40c7-b286-0d1b9706aec6-env-overrides\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710037    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e716b8d-7461-4f3f-b8e2-63a243537324-cni-binary-copy\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710058    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-run-multus-certs\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.711010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710084    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-etc-selinux\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710111    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-sysctl-d\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710126    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a690986-5200-4f8f-803d-1db7de256ddb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710215    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-etc-selinux\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710307    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-sysctl-d\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710308    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0281d37-6c54-407c-bc65-61d01eeb63b9-iptables-alerter-script\") pod \"iptables-alerter-bnnxp\" (UID: \"a0281d37-6c54-407c-bc65-61d01eeb63b9\") " pod="openshift-network-operator/iptables-alerter-bnnxp"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710135    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-lib-modules\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710328    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-lib-modules\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710340    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-tuned\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710364    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-run-netns\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710378    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-run-multus-certs\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710389    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvklb\" (UniqueName: \"kubernetes.io/projected/1e716b8d-7461-4f3f-b8e2-63a243537324-kube-api-access-cvklb\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710407    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a690986-5200-4f8f-803d-1db7de256ddb-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710483    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-run-netns\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.711763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710532    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f542j\" (UniqueName: \"kubernetes.io/projected/a0281d37-6c54-407c-bc65-61d01eeb63b9-kube-api-access-f542j\") pod \"iptables-alerter-bnnxp\" (UID: \"a0281d37-6c54-407c-bc65-61d01eeb63b9\") " pod="openshift-network-operator/iptables-alerter-bnnxp"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710583    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6c1dc49-8503-40c7-b286-0d1b9706aec6-env-overrides\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710584    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a690986-5200-4f8f-803d-1db7de256ddb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710665    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-etc-openvswitch\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710694    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e90e2983-dbaa-4989-8e2d-9e00a858af9e-hosts-file\") pod \"node-resolver-tgwm7\" (UID: \"e90e2983-dbaa-4989-8e2d-9e00a858af9e\") " pod="openshift-dns/node-resolver-tgwm7"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710698    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6c1dc49-8503-40c7-b286-0d1b9706aec6-ovnkube-config\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710722    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-etc-openvswitch\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710676    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710724    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3c283f9-edab-4267-ad1b-42b1f174c884-host\") pod \"node-ca-6fbtz\" (UID: \"c3c283f9-edab-4267-ad1b-42b1f174c884\") " pod="openshift-image-registry/node-ca-6fbtz"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710700    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e716b8d-7461-4f3f-b8e2-63a243537324-cni-binary-copy\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710820    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-kubelet-config\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710848    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxlp4\" (UniqueName: \"kubernetes.io/projected/8695fc30-cdfd-4a63-bf30-33cf14224955-kube-api-access-vxlp4\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710879    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-run-k8s-cni-cncf-io\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710910    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-registration-dir\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710930    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-run-k8s-cni-cncf-io\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.712500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710936    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-tmp\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710966    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a690986-5200-4f8f-803d-1db7de256ddb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.710979    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a690986-5200-4f8f-803d-1db7de256ddb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711012    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-registration-dir\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711016    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64637b8e-a4e3-44fc-bd3d-c69743d5bcfa-agent-certs\") pod \"konnectivity-agent-srlzs\" (UID: \"64637b8e-a4e3-44fc-bd3d-c69743d5bcfa\") " pod="kube-system/konnectivity-agent-srlzs"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711056    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-system-cni-dir\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711082    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e716b8d-7461-4f3f-b8e2-63a243537324-multus-daemon-config\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711104    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-cni-netd\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711118    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a690986-5200-4f8f-803d-1db7de256ddb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711133    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-system-cni-dir\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711157    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6c1dc49-8503-40c7-b286-0d1b9706aec6-ovnkube-script-lib\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711179    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-cni-netd\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711182    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90e2983-dbaa-4989-8e2d-9e00a858af9e-tmp-dir\") pod \"node-resolver-tgwm7\" (UID: \"e90e2983-dbaa-4989-8e2d-9e00a858af9e\") " pod="openshift-dns/node-resolver-tgwm7"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711209    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:24.713264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711236    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmg9x\" (UniqueName: \"kubernetes.io/projected/423e031a-a902-482b-8cf0-050a08d3260d-kube-api-access-fmg9x\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711261    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-systemd\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711285    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-run\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711309    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9vxc\" (UniqueName: \"kubernetes.io/projected/8a690986-5200-4f8f-803d-1db7de256ddb-kube-api-access-s9vxc\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711337    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0281d37-6c54-407c-bc65-61d01eeb63b9-host-slash\") pod \"iptables-alerter-bnnxp\" (UID: \"a0281d37-6c54-407c-bc65-61d01eeb63b9\") " pod="openshift-network-operator/iptables-alerter-bnnxp"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711345    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-systemd\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711379    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-run-ovn\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711405    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3c283f9-edab-4267-ad1b-42b1f174c884-serviceca\") pod \"node-ca-6fbtz\" (UID: \"c3c283f9-edab-4267-ad1b-42b1f174c884\") " pod="openshift-image-registry/node-ca-6fbtz"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711430    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-multus-cni-dir\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711453    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-etc-kubernetes\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-sysctl-conf\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711506    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w4bn\" (UniqueName: \"kubernetes.io/projected/e90e2983-dbaa-4989-8e2d-9e00a858af9e-kube-api-access-9w4bn\") pod \"node-resolver-tgwm7\" (UID: \"e90e2983-dbaa-4989-8e2d-9e00a858af9e\") " pod="openshift-dns/node-resolver-tgwm7"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711530    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-run-ovn\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711549    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711576    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-cnibin\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.713984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711590    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-run\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711606    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-systemd-units\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711629    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-etc-kubernetes\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711687    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e716b8d-7461-4f3f-b8e2-63a243537324-multus-daemon-config\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711687    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-systemd-units\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711702    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6c1dc49-8503-40c7-b286-0d1b9706aec6-ovnkube-script-lib\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711743    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-multus-cni-dir\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711741    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0281d37-6c54-407c-bc65-61d01eeb63b9-host-slash\") pod \"iptables-alerter-bnnxp\" (UID: \"a0281d37-6c54-407c-bc65-61d01eeb63b9\") " pod="openshift-network-operator/iptables-alerter-bnnxp"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711779    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-cnibin\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711937    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-sysctl-conf\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.711975    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-node-log\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712000    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-log-socket\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712024    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-var-lib-kubelet\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712051    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxrtp\" (UniqueName: \"kubernetes.io/projected/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-kube-api-access-mxrtp\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712076    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjgp\" (UniqueName: \"kubernetes.io/projected/a6c1dc49-8503-40c7-b286-0d1b9706aec6-kube-api-access-lhjgp\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.714528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712090    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-log-socket\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712098    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-run-netns\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712122    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-socket-dir\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712161    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-modprobe-d\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712186    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6c1dc49-8503-40c7-b286-0d1b9706aec6-ovn-node-metrics-cert\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712212    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jgt6\" (UniqueName: \"kubernetes.io/projected/c3c283f9-edab-4267-ad1b-42b1f174c884-kube-api-access-4jgt6\") pod \"node-ca-6fbtz\" (UID: \"c3c283f9-edab-4267-ad1b-42b1f174c884\") " pod="openshift-image-registry/node-ca-6fbtz"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712225    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-var-lib-kubelet\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712236    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64637b8e-a4e3-44fc-bd3d-c69743d5bcfa-konnectivity-ca\") pod \"konnectivity-agent-srlzs\" (UID: \"64637b8e-a4e3-44fc-bd3d-c69743d5bcfa\") " pod="kube-system/konnectivity-agent-srlzs"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712265    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-multus-conf-dir\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712290    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a690986-5200-4f8f-803d-1db7de256ddb-system-cni-dir\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712331    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-run-systemd\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712344    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-multus-conf-dir\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712050    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-node-log\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712333    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/423e031a-a902-482b-8cf0-050a08d3260d-socket-dir\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712410    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e716b8d-7461-4f3f-b8e2-63a243537324-host-run-netns\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.715043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712402    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-run-systemd\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.715470 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712440    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-modprobe-d\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.715470 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712483    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a690986-5200-4f8f-803d-1db7de256ddb-system-cni-dir\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.715470 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712522    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-run-ovn-kubernetes\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.715470 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712549    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-kubelet\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.715470 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712635    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-run-ovn-kubernetes\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.715470 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.712638    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6c1dc49-8503-40c7-b286-0d1b9706aec6-host-kubelet\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.715470 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.714280    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-etc-tuned\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.715470 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.714410    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-tmp\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.715470 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.714694    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6c1dc49-8503-40c7-b286-0d1b9706aec6-ovn-node-metrics-cert\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.720507 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.720487    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvklb\" (UniqueName: \"kubernetes.io/projected/1e716b8d-7461-4f3f-b8e2-63a243537324-kube-api-access-cvklb\") pod \"multus-78dlw\" (UID: \"1e716b8d-7461-4f3f-b8e2-63a243537324\") " pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.722814 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.722788    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f542j\" (UniqueName: \"kubernetes.io/projected/a0281d37-6c54-407c-bc65-61d01eeb63b9-kube-api-access-f542j\") pod \"iptables-alerter-bnnxp\" (UID: \"a0281d37-6c54-407c-bc65-61d01eeb63b9\") " pod="openshift-network-operator/iptables-alerter-bnnxp"
Jul 09 18:04:24.723995 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.723974    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjgp\" (UniqueName: \"kubernetes.io/projected/a6c1dc49-8503-40c7-b286-0d1b9706aec6-kube-api-access-lhjgp\") pod \"ovnkube-node-hmxsn\" (UID: \"a6c1dc49-8503-40c7-b286-0d1b9706aec6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.723995 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.723985    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmg9x\" (UniqueName: \"kubernetes.io/projected/423e031a-a902-482b-8cf0-050a08d3260d-kube-api-access-fmg9x\") pod \"aws-ebs-csi-driver-node-k4h2n\" (UID: \"423e031a-a902-482b-8cf0-050a08d3260d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.724518 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.724491    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9vxc\" (UniqueName: \"kubernetes.io/projected/8a690986-5200-4f8f-803d-1db7de256ddb-kube-api-access-s9vxc\") pod \"multus-additional-cni-plugins-t9r9n\" (UID: \"8a690986-5200-4f8f-803d-1db7de256ddb\") " pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:24.724622 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.724600    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxrtp\" (UniqueName: \"kubernetes.io/projected/3ea667e0-903b-43e7-b192-0ba95ac6c1f9-kube-api-access-mxrtp\") pod \"tuned-hg7xk\" (UID: \"3ea667e0-903b-43e7-b192-0ba95ac6c1f9\") " pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.813429 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813383    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jgt6\" (UniqueName: \"kubernetes.io/projected/c3c283f9-edab-4267-ad1b-42b1f174c884-kube-api-access-4jgt6\") pod \"node-ca-6fbtz\" (UID: \"c3c283f9-edab-4267-ad1b-42b1f174c884\") " pod="openshift-image-registry/node-ca-6fbtz"
Jul 09 18:04:24.813429 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813427    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64637b8e-a4e3-44fc-bd3d-c69743d5bcfa-konnectivity-ca\") pod \"konnectivity-agent-srlzs\" (UID: \"64637b8e-a4e3-44fc-bd3d-c69743d5bcfa\") " pod="kube-system/konnectivity-agent-srlzs"
Jul 09 18:04:24.813685 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813452    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-dbus\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:24.813685 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813481    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5jk\" (UniqueName: \"kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk\") pod \"network-check-target-c9jnd\" (UID: \"d50047de-b0e9-4a3c-99ad-b4566e13172c\") " pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:24.813685 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813585    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e90e2983-dbaa-4989-8e2d-9e00a858af9e-hosts-file\") pod \"node-resolver-tgwm7\" (UID: \"e90e2983-dbaa-4989-8e2d-9e00a858af9e\") " pod="openshift-dns/node-resolver-tgwm7"
Jul 09 18:04:24.813685 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813614    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3c283f9-edab-4267-ad1b-42b1f174c884-host\") pod \"node-ca-6fbtz\" (UID: \"c3c283f9-edab-4267-ad1b-42b1f174c884\") " pod="openshift-image-registry/node-ca-6fbtz"
Jul 09 18:04:24.813685 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813611    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-dbus\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:24.813685 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813639    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-kubelet-config\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:24.813685 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813666    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxlp4\" (UniqueName: \"kubernetes.io/projected/8695fc30-cdfd-4a63-bf30-33cf14224955-kube-api-access-vxlp4\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:24.813685 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813674    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3c283f9-edab-4267-ad1b-42b1f174c884-host\") pod \"node-ca-6fbtz\" (UID: \"c3c283f9-edab-4267-ad1b-42b1f174c884\") " pod="openshift-image-registry/node-ca-6fbtz"
Jul 09 18:04:24.814020 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813697    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64637b8e-a4e3-44fc-bd3d-c69743d5bcfa-agent-certs\") pod \"konnectivity-agent-srlzs\" (UID: \"64637b8e-a4e3-44fc-bd3d-c69743d5bcfa\") " pod="kube-system/konnectivity-agent-srlzs"
Jul 09 18:04:24.814020 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813703    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e90e2983-dbaa-4989-8e2d-9e00a858af9e-hosts-file\") pod \"node-resolver-tgwm7\" (UID: \"e90e2983-dbaa-4989-8e2d-9e00a858af9e\") " pod="openshift-dns/node-resolver-tgwm7"
Jul 09 18:04:24.814020 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813725    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90e2983-dbaa-4989-8e2d-9e00a858af9e-tmp-dir\") pod \"node-resolver-tgwm7\" (UID: \"e90e2983-dbaa-4989-8e2d-9e00a858af9e\") " pod="openshift-dns/node-resolver-tgwm7"
Jul 09 18:04:24.814020 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813738    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-kubelet-config\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:24.814020 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813749    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:24.814020 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813779    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3c283f9-edab-4267-ad1b-42b1f174c884-serviceca\") pod \"node-ca-6fbtz\" (UID: \"c3c283f9-edab-4267-ad1b-42b1f174c884\") " pod="openshift-image-registry/node-ca-6fbtz"
Jul 09 18:04:24.814020 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813805    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w4bn\" (UniqueName: \"kubernetes.io/projected/e90e2983-dbaa-4989-8e2d-9e00a858af9e-kube-api-access-9w4bn\") pod \"node-resolver-tgwm7\" (UID: \"e90e2983-dbaa-4989-8e2d-9e00a858af9e\") " pod="openshift-dns/node-resolver-tgwm7"
Jul 09 18:04:24.814020 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.813833    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:24.814020 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.813931    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:24.814020 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.813929    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:24.814478 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.814035    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret podName:f0f42946-27c2-47be-b09b-cefcd2f8ce8e nodeName:}" failed. No retries permitted until 2026-07-09 18:04:25.313987002 +0000 UTC m=+2.152329988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret") pod "global-pull-secret-syncer-j6pks" (UID: "f0f42946-27c2-47be-b09b-cefcd2f8ce8e") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:24.814478 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.814069    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90e2983-dbaa-4989-8e2d-9e00a858af9e-tmp-dir\") pod \"node-resolver-tgwm7\" (UID: \"e90e2983-dbaa-4989-8e2d-9e00a858af9e\") " pod="openshift-dns/node-resolver-tgwm7"
Jul 09 18:04:24.814478 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.814087    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs podName:8695fc30-cdfd-4a63-bf30-33cf14224955 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:25.314064992 +0000 UTC m=+2.152407957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs") pod "network-metrics-daemon-pw25w" (UID: "8695fc30-cdfd-4a63-bf30-33cf14224955") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:24.814478 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.814141    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64637b8e-a4e3-44fc-bd3d-c69743d5bcfa-konnectivity-ca\") pod \"konnectivity-agent-srlzs\" (UID: \"64637b8e-a4e3-44fc-bd3d-c69743d5bcfa\") " pod="kube-system/konnectivity-agent-srlzs"
Jul 09 18:04:24.814478 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.814372    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3c283f9-edab-4267-ad1b-42b1f174c884-serviceca\") pod \"node-ca-6fbtz\" (UID: \"c3c283f9-edab-4267-ad1b-42b1f174c884\") " pod="openshift-image-registry/node-ca-6fbtz"
Jul 09 18:04:24.816174 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.816154    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64637b8e-a4e3-44fc-bd3d-c69743d5bcfa-agent-certs\") pod \"konnectivity-agent-srlzs\" (UID: \"64637b8e-a4e3-44fc-bd3d-c69743d5bcfa\") " pod="kube-system/konnectivity-agent-srlzs"
Jul 09 18:04:24.827570 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.827547    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w4bn\" (UniqueName: \"kubernetes.io/projected/e90e2983-dbaa-4989-8e2d-9e00a858af9e-kube-api-access-9w4bn\") pod \"node-resolver-tgwm7\" (UID: \"e90e2983-dbaa-4989-8e2d-9e00a858af9e\") " pod="openshift-dns/node-resolver-tgwm7"
Jul 09 18:04:24.829328 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.829307    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxlp4\" (UniqueName: \"kubernetes.io/projected/8695fc30-cdfd-4a63-bf30-33cf14224955-kube-api-access-vxlp4\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:24.832375 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.832359    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:24.832425 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.832378    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:24.832425 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.832388    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cf5jk for pod openshift-network-diagnostics/network-check-target-c9jnd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:24.832493 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:24.832455    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk podName:d50047de-b0e9-4a3c-99ad-b4566e13172c nodeName:}" failed. No retries permitted until 2026-07-09 18:04:25.332441589 +0000 UTC m=+2.170784556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cf5jk" (UniqueName: "kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk") pod "network-check-target-c9jnd" (UID: "d50047de-b0e9-4a3c-99ad-b4566e13172c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:24.834221 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.834203    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jgt6\" (UniqueName: \"kubernetes.io/projected/c3c283f9-edab-4267-ad1b-42b1f174c884-kube-api-access-4jgt6\") pod \"node-ca-6fbtz\" (UID: \"c3c283f9-edab-4267-ad1b-42b1f174c884\") " pod="openshift-image-registry/node-ca-6fbtz"
Jul 09 18:04:24.931574 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.931535    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:24.937602 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.937583    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-78dlw"
Jul 09 18:04:24.950376 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.950353    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n"
Jul 09 18:04:24.967597 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.967557    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hg7xk"
Jul 09 18:04:24.986952 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:24.986921    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t9r9n"
Jul 09 18:04:25.002772 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.002747    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bnnxp"
Jul 09 18:04:25.015141 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.015118    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-srlzs"
Jul 09 18:04:25.035487 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.035463    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tgwm7"
Jul 09 18:04:25.038163 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.038147    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6fbtz"
Jul 09 18:04:25.264619 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:25.264582    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54f981040f96dfb2de508fe0606b9e4.slice/crio-9d5e9c5ef77a3082fbbc228e63245bf9ae01e0a9e5abbe1eddef8e6703922558 WatchSource:0}: Error finding container 9d5e9c5ef77a3082fbbc228e63245bf9ae01e0a9e5abbe1eddef8e6703922558: Status 404 returned error can't find the container with id 9d5e9c5ef77a3082fbbc228e63245bf9ae01e0a9e5abbe1eddef8e6703922558
Jul 09 18:04:25.265074 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:25.265041    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ef526d198a51a2dc5e772145e24829.slice/crio-bf0672d346ca0c37f4b49d38a0cabd269132839bf5152e31437add04cbdf2450 WatchSource:0}: Error finding container bf0672d346ca0c37f4b49d38a0cabd269132839bf5152e31437add04cbdf2450: Status 404 returned error can't find the container with id bf0672d346ca0c37f4b49d38a0cabd269132839bf5152e31437add04cbdf2450
Jul 09 18:04:25.271024 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.271008    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:04:25.317114 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.317078    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:25.317263 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:25.317208    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:25.317263 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.317222    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:25.317370 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:25.317271    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs podName:8695fc30-cdfd-4a63-bf30-33cf14224955 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:26.317249533 +0000 UTC m=+3.155592523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs") pod "network-metrics-daemon-pw25w" (UID: "8695fc30-cdfd-4a63-bf30-33cf14224955") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:25.317370 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:25.317294    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:25.317370 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:25.317341    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret podName:f0f42946-27c2-47be-b09b-cefcd2f8ce8e nodeName:}" failed. No retries permitted until 2026-07-09 18:04:26.317328756 +0000 UTC m=+3.155671720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret") pod "global-pull-secret-syncer-j6pks" (UID: "f0f42946-27c2-47be-b09b-cefcd2f8ce8e") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:25.352053 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:25.352028    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423e031a_a902_482b_8cf0_050a08d3260d.slice/crio-bcf1604521484a99d8c9f3a868cd9acfaa71d37734b2672dd92d76f149731b3f WatchSource:0}: Error finding container bcf1604521484a99d8c9f3a868cd9acfaa71d37734b2672dd92d76f149731b3f: Status 404 returned error can't find the container with id bcf1604521484a99d8c9f3a868cd9acfaa71d37734b2672dd92d76f149731b3f
Jul 09 18:04:25.383836 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:25.383808    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e716b8d_7461_4f3f_b8e2_63a243537324.slice/crio-e90d673c6910f814c7cca1e63171bf25eebe359c56bddaa94c48ecc0716a6187 WatchSource:0}: Error finding container e90d673c6910f814c7cca1e63171bf25eebe359c56bddaa94c48ecc0716a6187: Status 404 returned error can't find the container with id e90d673c6910f814c7cca1e63171bf25eebe359c56bddaa94c48ecc0716a6187
Jul 09 18:04:25.404510 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:25.404486    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90e2983_dbaa_4989_8e2d_9e00a858af9e.slice/crio-0fc3715612e31ea4d4861cbc386a4113df5249f5dcb69f3508e8a66042af5a51 WatchSource:0}: Error finding container 0fc3715612e31ea4d4861cbc386a4113df5249f5dcb69f3508e8a66042af5a51: Status 404 returned error can't find the container with id 0fc3715612e31ea4d4861cbc386a4113df5249f5dcb69f3508e8a66042af5a51
Jul 09 18:04:25.417934 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.417910    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5jk\" (UniqueName: \"kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk\") pod \"network-check-target-c9jnd\" (UID: \"d50047de-b0e9-4a3c-99ad-b4566e13172c\") " pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:25.418073 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:25.418059    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:25.418145 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:25.418079    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:25.418145 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:25.418088    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cf5jk for pod openshift-network-diagnostics/network-check-target-c9jnd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:25.418145 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:25.418144    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk podName:d50047de-b0e9-4a3c-99ad-b4566e13172c nodeName:}" failed. No retries permitted until 2026-07-09 18:04:26.418117817 +0000 UTC m=+3.256460781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf5jk" (UniqueName: "kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk") pod "network-check-target-c9jnd" (UID: "d50047de-b0e9-4a3c-99ad-b4566e13172c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:25.462181 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:25.462151    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c1dc49_8503_40c7_b286_0d1b9706aec6.slice/crio-3b0199f7ecb59a8ed21d4146595b707f10f32e5d02d0eb4d2d58decdd1730dc8 WatchSource:0}: Error finding container 3b0199f7ecb59a8ed21d4146595b707f10f32e5d02d0eb4d2d58decdd1730dc8: Status 404 returned error can't find the container with id 3b0199f7ecb59a8ed21d4146595b707f10f32e5d02d0eb4d2d58decdd1730dc8
Jul 09 18:04:25.494094 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:25.494066    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0281d37_6c54_407c_bc65_61d01eeb63b9.slice/crio-1fab86711ab590a9183012d152c65814821c56423b175d304c9ef6e859701065 WatchSource:0}: Error finding container 1fab86711ab590a9183012d152c65814821c56423b175d304c9ef6e859701065: Status 404 returned error can't find the container with id 1fab86711ab590a9183012d152c65814821c56423b175d304c9ef6e859701065
Jul 09 18:04:25.495111 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:25.495084    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c283f9_edab_4267_ad1b_42b1f174c884.slice/crio-ab360c30ca0422fd98533ba49a14bd75365afb513da2ef4f69a316c0bb7c33a5 WatchSource:0}: Error finding container ab360c30ca0422fd98533ba49a14bd75365afb513da2ef4f69a316c0bb7c33a5: Status 404 returned error can't find the container with id ab360c30ca0422fd98533ba49a14bd75365afb513da2ef4f69a316c0bb7c33a5
Jul 09 18:04:25.543333 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:25.543302    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64637b8e_a4e3_44fc_bd3d_c69743d5bcfa.slice/crio-432c2626aef573d280177993e4a5601b9014dc16857813ae86f1c7a73f046881 WatchSource:0}: Error finding container 432c2626aef573d280177993e4a5601b9014dc16857813ae86f1c7a73f046881: Status 404 returned error can't find the container with id 432c2626aef573d280177993e4a5601b9014dc16857813ae86f1c7a73f046881
Jul 09 18:04:25.571564 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.571534    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:25.595213 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:25.595185    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a690986_5200_4f8f_803d_1db7de256ddb.slice/crio-3713e4e37fa2ee4a7d256d2b09b47ef4b02b8546ae1a2c893271058572aba3ff WatchSource:0}: Error finding container 3713e4e37fa2ee4a7d256d2b09b47ef4b02b8546ae1a2c893271058572aba3ff: Status 404 returned error can't find the container with id 3713e4e37fa2ee4a7d256d2b09b47ef4b02b8546ae1a2c893271058572aba3ff
Jul 09 18:04:25.643414 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:04:25.643386    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea667e0_903b_43e7_b192_0ba95ac6c1f9.slice/crio-6da19685ed3a7a224e01f728fcc1938864ce97be331307442eac0039310de3b3 WatchSource:0}: Error finding container 6da19685ed3a7a224e01f728fcc1938864ce97be331307442eac0039310de3b3: Status 404 returned error can't find the container with id 6da19685ed3a7a224e01f728fcc1938864ce97be331307442eac0039310de3b3
Jul 09 18:04:25.663798 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.663765    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:25 +0000 UTC" deadline="2028-03-26 23:57:18.193693282 +0000 UTC"
Jul 09 18:04:25.663932 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.663795    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15029h52m52.529902436s"
Jul 09 18:04:25.928491 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.928407    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:25.928684 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:25.928552    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:25.929600 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.929004    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:25.929600 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:25.929103    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:25.936913 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.936860    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hg7xk" event={"ID":"3ea667e0-903b-43e7-b192-0ba95ac6c1f9","Type":"ContainerStarted","Data":"6da19685ed3a7a224e01f728fcc1938864ce97be331307442eac0039310de3b3"}
Jul 09 18:04:25.939944 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.939799    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9r9n" event={"ID":"8a690986-5200-4f8f-803d-1db7de256ddb","Type":"ContainerStarted","Data":"3713e4e37fa2ee4a7d256d2b09b47ef4b02b8546ae1a2c893271058572aba3ff"}
Jul 09 18:04:25.942581 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.942506    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-srlzs" event={"ID":"64637b8e-a4e3-44fc-bd3d-c69743d5bcfa","Type":"ContainerStarted","Data":"432c2626aef573d280177993e4a5601b9014dc16857813ae86f1c7a73f046881"}
Jul 09 18:04:25.950138 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.950045    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bnnxp" event={"ID":"a0281d37-6c54-407c-bc65-61d01eeb63b9","Type":"ContainerStarted","Data":"1fab86711ab590a9183012d152c65814821c56423b175d304c9ef6e859701065"}
Jul 09 18:04:25.951751 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.951672    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tgwm7" event={"ID":"e90e2983-dbaa-4989-8e2d-9e00a858af9e","Type":"ContainerStarted","Data":"0fc3715612e31ea4d4861cbc386a4113df5249f5dcb69f3508e8a66042af5a51"}
Jul 09 18:04:25.955100 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.955034    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-78dlw" event={"ID":"1e716b8d-7461-4f3f-b8e2-63a243537324","Type":"ContainerStarted","Data":"e90d673c6910f814c7cca1e63171bf25eebe359c56bddaa94c48ecc0716a6187"}
Jul 09 18:04:25.957211 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.957184    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n" event={"ID":"423e031a-a902-482b-8cf0-050a08d3260d","Type":"ContainerStarted","Data":"bcf1604521484a99d8c9f3a868cd9acfaa71d37734b2672dd92d76f149731b3f"}
Jul 09 18:04:25.959383 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.959356    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" event={"ID":"09ef526d198a51a2dc5e772145e24829","Type":"ContainerStarted","Data":"bf0672d346ca0c37f4b49d38a0cabd269132839bf5152e31437add04cbdf2450"}
Jul 09 18:04:25.961961 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.961935    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6fbtz" event={"ID":"c3c283f9-edab-4267-ad1b-42b1f174c884","Type":"ContainerStarted","Data":"ab360c30ca0422fd98533ba49a14bd75365afb513da2ef4f69a316c0bb7c33a5"}
Jul 09 18:04:25.964086 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.964058    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn" event={"ID":"a6c1dc49-8503-40c7-b286-0d1b9706aec6","Type":"ContainerStarted","Data":"3b0199f7ecb59a8ed21d4146595b707f10f32e5d02d0eb4d2d58decdd1730dc8"}
Jul 09 18:04:25.965810 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:25.965782    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal" event={"ID":"c54f981040f96dfb2de508fe0606b9e4","Type":"ContainerStarted","Data":"9d5e9c5ef77a3082fbbc228e63245bf9ae01e0a9e5abbe1eddef8e6703922558"}
Jul 09 18:04:26.327577 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:26.326470    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:26.327577 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:26.326541    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:26.327577 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:26.326764    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:26.327577 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:26.326834    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret podName:f0f42946-27c2-47be-b09b-cefcd2f8ce8e nodeName:}" failed. No retries permitted until 2026-07-09 18:04:28.326813399 +0000 UTC m=+5.165156377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret") pod "global-pull-secret-syncer-j6pks" (UID: "f0f42946-27c2-47be-b09b-cefcd2f8ce8e") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:26.327577 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:26.327353    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:26.327577 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:26.327406    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs podName:8695fc30-cdfd-4a63-bf30-33cf14224955 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:28.327389209 +0000 UTC m=+5.165732179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs") pod "network-metrics-daemon-pw25w" (UID: "8695fc30-cdfd-4a63-bf30-33cf14224955") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:26.427751 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:26.427712    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5jk\" (UniqueName: \"kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk\") pod \"network-check-target-c9jnd\" (UID: \"d50047de-b0e9-4a3c-99ad-b4566e13172c\") " pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:26.427958 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:26.427938    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:26.427958 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:26.427959    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:26.428044 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:26.427973    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cf5jk for pod openshift-network-diagnostics/network-check-target-c9jnd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:26.428044 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:26.428036    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk podName:d50047de-b0e9-4a3c-99ad-b4566e13172c nodeName:}" failed. No retries permitted until 2026-07-09 18:04:28.428016442 +0000 UTC m=+5.266359424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf5jk" (UniqueName: "kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk") pod "network-check-target-c9jnd" (UID: "d50047de-b0e9-4a3c-99ad-b4566e13172c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:26.607335 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:26.607253    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:26.664805 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:26.664740    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:25 +0000 UTC" deadline="2028-03-09 03:15:39.992486375 +0000 UTC"
Jul 09 18:04:26.664805 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:26.664803    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14601h11m13.327687599s"
Jul 09 18:04:26.928202 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:26.928077    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:26.928410 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:26.928253    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:27.937999 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:27.935786    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:27.937999 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:27.935908    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:27.937999 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:27.935978    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:27.937999 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:27.936060    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:28.344059 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:28.343965    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:28.344059 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:28.344015    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:28.344288 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:28.344158    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:28.344288 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:28.344221    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret podName:f0f42946-27c2-47be-b09b-cefcd2f8ce8e nodeName:}" failed. No retries permitted until 2026-07-09 18:04:32.344199907 +0000 UTC m=+9.182542891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret") pod "global-pull-secret-syncer-j6pks" (UID: "f0f42946-27c2-47be-b09b-cefcd2f8ce8e") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:28.344672 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:28.344652    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:28.344773 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:28.344714    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs podName:8695fc30-cdfd-4a63-bf30-33cf14224955 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:32.344697348 +0000 UTC m=+9.183040315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs") pod "network-metrics-daemon-pw25w" (UID: "8695fc30-cdfd-4a63-bf30-33cf14224955") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:28.445521 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:28.445479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5jk\" (UniqueName: \"kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk\") pod \"network-check-target-c9jnd\" (UID: \"d50047de-b0e9-4a3c-99ad-b4566e13172c\") " pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:28.445737 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:28.445695    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:28.445737 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:28.445721    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:28.445737 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:28.445735    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cf5jk for pod openshift-network-diagnostics/network-check-target-c9jnd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:28.445919 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:28.445797    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk podName:d50047de-b0e9-4a3c-99ad-b4566e13172c nodeName:}" failed. No retries permitted until 2026-07-09 18:04:32.445776237 +0000 UTC m=+9.284119203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf5jk" (UniqueName: "kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk") pod "network-check-target-c9jnd" (UID: "d50047de-b0e9-4a3c-99ad-b4566e13172c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:28.928581 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:28.928084    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:28.928581 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:28.928203    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:29.934062 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:29.933527    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:29.934062 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:29.933693    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:29.934062 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:29.933717    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:29.934062 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:29.933845    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:30.928119 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:30.928076    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:30.928330 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:30.928227    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:31.927700 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:31.927663    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:31.928179 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:31.927830    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:31.928292 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:31.928276    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:31.928411 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:31.928382    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:32.383219 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:32.383128    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:32.383219 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:32.383184    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:32.383463 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:32.383317    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:32.383463 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:32.383336    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:32.383463 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:32.383402    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret podName:f0f42946-27c2-47be-b09b-cefcd2f8ce8e nodeName:}" failed. No retries permitted until 2026-07-09 18:04:40.383383425 +0000 UTC m=+17.221726392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret") pod "global-pull-secret-syncer-j6pks" (UID: "f0f42946-27c2-47be-b09b-cefcd2f8ce8e") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:32.383463 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:32.383425    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs podName:8695fc30-cdfd-4a63-bf30-33cf14224955 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:40.383413597 +0000 UTC m=+17.221756561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs") pod "network-metrics-daemon-pw25w" (UID: "8695fc30-cdfd-4a63-bf30-33cf14224955") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:32.484284 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:32.484241    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5jk\" (UniqueName: \"kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk\") pod \"network-check-target-c9jnd\" (UID: \"d50047de-b0e9-4a3c-99ad-b4566e13172c\") " pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:32.484453 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:32.484421    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:32.484453 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:32.484441    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:32.484570 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:32.484455    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cf5jk for pod openshift-network-diagnostics/network-check-target-c9jnd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:32.484570 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:32.484527    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk podName:d50047de-b0e9-4a3c-99ad-b4566e13172c nodeName:}" failed. No retries permitted until 2026-07-09 18:04:40.484506943 +0000 UTC m=+17.322849921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf5jk" (UniqueName: "kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk") pod "network-check-target-c9jnd" (UID: "d50047de-b0e9-4a3c-99ad-b4566e13172c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:32.928156 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:32.928119    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:32.928697 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:32.928272    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:33.932685 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:33.931993    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:33.932685 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:33.932144    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:33.932685 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:33.932532    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:33.932685 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:33.932623    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:34.927998 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:34.927955    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:34.928208 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:34.928108    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:35.928933 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:35.928699    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:35.929332 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:35.929041    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:35.929542 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:35.929516    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:35.929687 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:35.929662    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:36.927594 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:36.927544    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:36.927768 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:36.927674    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:37.927821 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:37.927786    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:37.928181 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:37.927917    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:37.928181 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:37.927964    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:37.928181 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:37.928049    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:38.927832 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:38.927793    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:38.928280 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:38.927919    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:39.928089 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:39.928046    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:39.928548 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:39.928206    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:39.928548 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:39.928288    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:39.928548 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:39.928374    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:40.444003 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:40.443964    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:40.444175 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:40.444015    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:40.444175 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:40.444107    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:40.444175 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:40.444118    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:40.444281 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:40.444184    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs podName:8695fc30-cdfd-4a63-bf30-33cf14224955 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:56.444167605 +0000 UTC m=+33.282510568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs") pod "network-metrics-daemon-pw25w" (UID: "8695fc30-cdfd-4a63-bf30-33cf14224955") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:40.444281 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:40.444220    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret podName:f0f42946-27c2-47be-b09b-cefcd2f8ce8e nodeName:}" failed. No retries permitted until 2026-07-09 18:04:56.444203649 +0000 UTC m=+33.282546615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret") pod "global-pull-secret-syncer-j6pks" (UID: "f0f42946-27c2-47be-b09b-cefcd2f8ce8e") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:40.545237 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:40.545204    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5jk\" (UniqueName: \"kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk\") pod \"network-check-target-c9jnd\" (UID: \"d50047de-b0e9-4a3c-99ad-b4566e13172c\") " pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:40.545382 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:40.545364    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:40.545425 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:40.545388    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:40.545425 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:40.545399    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cf5jk for pod openshift-network-diagnostics/network-check-target-c9jnd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:40.545494 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:40.545445    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk podName:d50047de-b0e9-4a3c-99ad-b4566e13172c nodeName:}" failed. No retries permitted until 2026-07-09 18:04:56.545431809 +0000 UTC m=+33.383774770 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf5jk" (UniqueName: "kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk") pod "network-check-target-c9jnd" (UID: "d50047de-b0e9-4a3c-99ad-b4566e13172c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:40.928393 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:40.928314    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:40.928857 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:40.928426    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:41.927990 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:41.927960    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:41.928153 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:41.928073    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:41.928153 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:41.928103    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:41.928261 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:41.928160    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:42.928348 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:42.928320    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:42.928734 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:42.928413    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:43.929003 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:43.928969    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:43.929438 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:43.929017    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:43.929438 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:43.929093    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:43.929438 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:43.929239    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:44.927761 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:44.927728    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:44.927924 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:44.927868    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:45.927755 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:45.927725    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:45.928187 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:45.927725    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:45.928187 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:45.927852    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:45.928187 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:45.927920    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:46.928539 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:46.928514    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:46.928833 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:46.928654    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:47.021814 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.021591    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hg7xk" event={"ID":"3ea667e0-903b-43e7-b192-0ba95ac6c1f9","Type":"ContainerStarted","Data":"5422aec5e3fc6f851024cd3c23e7d56670ee251910b4e8b6af687b34f3a8bc14"}
Jul 09 18:04:47.023776 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.023744    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-78dlw" event={"ID":"1e716b8d-7461-4f3f-b8e2-63a243537324","Type":"ContainerStarted","Data":"81174f24825d2f632747c630d8b7d0d09a8280bb17c267408467cc5483559d74"}
Jul 09 18:04:47.027280 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.027259    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:04:47.027629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.027608    2572 generic.go:356] "Generic (PLEG): container finished" podID="a6c1dc49-8503-40c7-b286-0d1b9706aec6" containerID="9fa3c52789fef5055b4e0afb6e0a81ada6d1c3a5f2beec020c19c95868d3cc88" exitCode=1
Jul 09 18:04:47.027716 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.027674    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn" event={"ID":"a6c1dc49-8503-40c7-b286-0d1b9706aec6","Type":"ContainerStarted","Data":"5b9ce199e9dd88f38963bd824daaf00d9c6519d171fc9feaba79c2a96256cab7"}
Jul 09 18:04:47.027773 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.027752    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn" event={"ID":"a6c1dc49-8503-40c7-b286-0d1b9706aec6","Type":"ContainerStarted","Data":"3d49e528cfc4bb3d11f9f4afaf166da67d3d321b0ee3005b3083592527ce8cd7"}
Jul 09 18:04:47.027841 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.027824    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn" event={"ID":"a6c1dc49-8503-40c7-b286-0d1b9706aec6","Type":"ContainerStarted","Data":"cacff946a4407425231096cdb8fdb732a3ab0afd3bf61c3a063275a3b27790bc"}
Jul 09 18:04:47.027896 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.027877    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn" event={"ID":"a6c1dc49-8503-40c7-b286-0d1b9706aec6","Type":"ContainerDied","Data":"9fa3c52789fef5055b4e0afb6e0a81ada6d1c3a5f2beec020c19c95868d3cc88"}
Jul 09 18:04:47.027948 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.027927    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn" event={"ID":"a6c1dc49-8503-40c7-b286-0d1b9706aec6","Type":"ContainerStarted","Data":"3a8ee29fa32c6fdca5e8a05e95d2d0e91f446fdec33b0587251f6d497d29c595"}
Jul 09 18:04:47.029292 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.029265    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal" event={"ID":"c54f981040f96dfb2de508fe0606b9e4","Type":"ContainerStarted","Data":"7d08aada783c9fba9128a045015627bab8414e29857a043e8cbadd50e998f3cc"}
Jul 09 18:04:47.057883 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.057836    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hg7xk" podStartSLOduration=2.388791308 podStartE2EDuration="23.057816388s" totalImagesPullingTime="20.66902508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:47.057384699 +0000 UTC m=+23.895727683" watchObservedRunningTime="2026-07-09 18:04:47.057816388 +0000 UTC m=+23.896159373"
Jul 09 18:04:47.122526 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.122480    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-78dlw" podStartSLOduration=2.177448655 podStartE2EDuration="23.122463631s" totalImagesPullingTime="20.945014976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:47.122449817 +0000 UTC m=+23.960792800" watchObservedRunningTime="2026-07-09 18:04:47.122463631 +0000 UTC m=+23.960806615"
Jul 09 18:04:47.122924 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.122891    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-107.ec2.internal" podStartSLOduration=22.122880517 podStartE2EDuration="22.122880517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:47.086410165 +0000 UTC m=+23.924753149" watchObservedRunningTime="2026-07-09 18:04:47.122880517 +0000 UTC m=+23.961223501"
Jul 09 18:04:47.928087 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.927896    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:47.928192 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:47.927880    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:47.928192 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:47.928130    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:47.928271 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:47.928204    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:48.033245 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.033213    2572 generic.go:356] "Generic (PLEG): container finished" podID="8a690986-5200-4f8f-803d-1db7de256ddb" containerID="b8cb2273c3ec8b0a3bf0a10d7378cf502c0fd310457765ea1af673a0d120c8c7" exitCode=0
Jul 09 18:04:48.034028 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.033295    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9r9n" event={"ID":"8a690986-5200-4f8f-803d-1db7de256ddb","Type":"ContainerDied","Data":"b8cb2273c3ec8b0a3bf0a10d7378cf502c0fd310457765ea1af673a0d120c8c7"}
Jul 09 18:04:48.034931 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.034912    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-srlzs" event={"ID":"64637b8e-a4e3-44fc-bd3d-c69743d5bcfa","Type":"ContainerStarted","Data":"78c565bb5fc3a63081fdc3ca5282e4621a974d2a8e9d254e9c8d60e74ee53054"}
Jul 09 18:04:48.037027 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.037003    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bnnxp" event={"ID":"a0281d37-6c54-407c-bc65-61d01eeb63b9","Type":"ContainerStarted","Data":"99ca4a277a053c747f23b017346ff051655886d9df2bf695c0313eb5eb040157"}
Jul 09 18:04:48.038509 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.038484    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tgwm7" event={"ID":"e90e2983-dbaa-4989-8e2d-9e00a858af9e","Type":"ContainerStarted","Data":"c57cc0b4504ce2983351223b1d52257564ab78de92fa2ac75ad6a9af114e7fd2"}
Jul 09 18:04:48.040380 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.040357    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n" event={"ID":"423e031a-a902-482b-8cf0-050a08d3260d","Type":"ContainerStarted","Data":"b0d23306bc1f13b0b592edbd6b3dbb8357bb95c98daeea6902ba7f9f1b75d73d"}
Jul 09 18:04:48.041924 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.041902    2572 generic.go:356] "Generic (PLEG): container finished" podID="09ef526d198a51a2dc5e772145e24829" containerID="1ebd4a966af844f505a8a6311fc012749dc6182232432704786859815edfb78d" exitCode=0
Jul 09 18:04:48.042010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.041977    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" event={"ID":"09ef526d198a51a2dc5e772145e24829","Type":"ContainerDied","Data":"1ebd4a966af844f505a8a6311fc012749dc6182232432704786859815edfb78d"}
Jul 09 18:04:48.043894 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.043872    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6fbtz" event={"ID":"c3c283f9-edab-4267-ad1b-42b1f174c884","Type":"ContainerStarted","Data":"1b085cc771b3eee285fb58478bd3e629b3344cf17c909975b5f0da28ca7d1c4b"}
Jul 09 18:04:48.046909 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.046894    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:04:48.047389 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.047374    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn" event={"ID":"a6c1dc49-8503-40c7-b286-0d1b9706aec6","Type":"ContainerStarted","Data":"cef2a36481d59a276d507f9c3cfbfb2f4e0f68a88773296234481d3e0c7365ba"}
Jul 09 18:04:48.110868 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.110827    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tgwm7" podStartSLOduration=3.222295085 podStartE2EDuration="24.110814081s" totalImagesPullingTime="20.888518996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:48.093784981 +0000 UTC m=+24.932127967" watchObservedRunningTime="2026-07-09 18:04:48.110814081 +0000 UTC m=+24.949157072"
Jul 09 18:04:48.130296 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.130262    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bnnxp" podStartSLOduration=3.314658369 podStartE2EDuration="24.130249714s" totalImagesPullingTime="20.815591345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:48.130049531 +0000 UTC m=+24.968392515" watchObservedRunningTime="2026-07-09 18:04:48.130249714 +0000 UTC m=+24.968592707"
Jul 09 18:04:48.130508 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.130490    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6fbtz" podStartSLOduration=3.3343372000000002 podStartE2EDuration="24.130483879s" totalImagesPullingTime="20.796146679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:48.111063718 +0000 UTC m=+24.949406701" watchObservedRunningTime="2026-07-09 18:04:48.130483879 +0000 UTC m=+24.968826870"
Jul 09 18:04:48.159869 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.159838    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-srlzs" podStartSLOduration=3.410607691 podStartE2EDuration="24.15982683s" totalImagesPullingTime="20.749219139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:48.159536458 +0000 UTC m=+24.997879444" watchObservedRunningTime="2026-07-09 18:04:48.15982683 +0000 UTC m=+24.998169793"
Jul 09 18:04:48.470616 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.470588    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-srlzs"
Jul 09 18:04:48.471114 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.471098    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-srlzs"
Jul 09 18:04:48.688289 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.688267    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:04:48.751948 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.751840    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:04:48.688283052Z","UUID":"3db6d1d0-dd5d-4aea-a5c3-7b2e4455b664","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:04:48.753030 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.753007    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:04:48.753104 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.753038    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:04:48.927623 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:48.927598    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:48.927776 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:48.927687    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:49.051674 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:49.051605    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n" event={"ID":"423e031a-a902-482b-8cf0-050a08d3260d","Type":"ContainerStarted","Data":"66b2c453188168860e63c2b0136387687add12a95688c4591d4d475584e7a553"}
Jul 09 18:04:49.053655 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:49.053589    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" event={"ID":"09ef526d198a51a2dc5e772145e24829","Type":"ContainerStarted","Data":"4739eb9558e2977c08cdf8bc1482791ad85eae396f256ec79ecc463a390cacf1"}
Jul 09 18:04:49.074208 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:49.074151    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-107.ec2.internal" podStartSLOduration=24.074134352 podStartE2EDuration="24.074134352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:49.073790692 +0000 UTC m=+25.912133676" watchObservedRunningTime="2026-07-09 18:04:49.074134352 +0000 UTC m=+25.912477337"
Jul 09 18:04:49.928525 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:49.928498    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:49.928525 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:49.928516    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:49.928758 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:49.928642    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:49.928837 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:49.928802    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:50.057822 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:50.057740    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n" event={"ID":"423e031a-a902-482b-8cf0-050a08d3260d","Type":"ContainerStarted","Data":"dfdabc598b974b13b7ccc4a473cafea7a42cd1513899adfef1b111d3b8022d90"}
Jul 09 18:04:50.060834 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:50.060804    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:04:50.061231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:50.061208    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 09 18:04:50.061231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:50.061223    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn" event={"ID":"a6c1dc49-8503-40c7-b286-0d1b9706aec6","Type":"ContainerStarted","Data":"600afd76ca5ecec5b628e99ce142e093a46d7200dcd13deda2c4a210f61fb5b5"}
Jul 09 18:04:50.077189 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:50.077146    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4h2n" podStartSLOduration=2.9130621039999998 podStartE2EDuration="26.077133079s" totalImagesPullingTime="23.164070975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:50.077043721 +0000 UTC m=+26.915386708" watchObservedRunningTime="2026-07-09 18:04:50.077133079 +0000 UTC m=+26.915476063"
Jul 09 18:04:50.927673 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:50.927649    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:50.927839 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:50.927744    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:51.928159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:51.927889    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:51.928958 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:51.928187    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:51.928958 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:51.927992    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:51.928958 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:51.928410    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:52.069157 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:52.069131    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:04:52.069548 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:52.069515    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn" event={"ID":"a6c1dc49-8503-40c7-b286-0d1b9706aec6","Type":"ContainerStarted","Data":"52cf744dede332f9ac280fd6e0bfc8b6d93474d6e16b44ec44bbccc0c196f7ef"}
Jul 09 18:04:52.069943 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:52.069919    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:52.070109 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:52.070091    2572 scope.go:117] "RemoveContainer" containerID="9fa3c52789fef5055b4e0afb6e0a81ada6d1c3a5f2beec020c19c95868d3cc88"
Jul 09 18:04:52.088007 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:52.087983    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:52.927694 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:52.927659    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:52.927968 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:52.927789    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:53.029049 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:53.029018    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-srlzs"
Jul 09 18:04:53.029612 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:53.029160    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 09 18:04:53.029673 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:53.029610    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-srlzs"
Jul 09 18:04:53.929093 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:53.928931    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:53.929310 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:53.929020    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:53.929310 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:53.929266    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:53.929310 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:53.929151    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:54.077102 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:54.077073    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:04:54.077501 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:54.077393    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn" event={"ID":"a6c1dc49-8503-40c7-b286-0d1b9706aec6","Type":"ContainerStarted","Data":"c4021ac17cf622245ed48d90d7d90856e26e2f6d2e4f7ead3d7e965c7e27a9d7"}
Jul 09 18:04:54.077573 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:54.077520    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:54.077632 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:54.077575    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:54.079207 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:54.079181    2572 generic.go:356] "Generic (PLEG): container finished" podID="8a690986-5200-4f8f-803d-1db7de256ddb" containerID="dae8517db88fae61d294bee765a8f998240df3a9b119872853a820cbde94e7bf" exitCode=0
Jul 09 18:04:54.079319 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:54.079210    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9r9n" event={"ID":"8a690986-5200-4f8f-803d-1db7de256ddb","Type":"ContainerDied","Data":"dae8517db88fae61d294bee765a8f998240df3a9b119872853a820cbde94e7bf"}
Jul 09 18:04:54.092650 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:54.092629    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:04:54.128757 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:54.128705    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn" podStartSLOduration=9.233856591 podStartE2EDuration="30.128688421s" totalImagesPullingTime="20.89483183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:54.128020539 +0000 UTC m=+30.966363533" watchObservedRunningTime="2026-07-09 18:04:54.128688421 +0000 UTC m=+30.967031405"
Jul 09 18:04:54.928201 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:54.928171    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:54.928352 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:54.928264    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:55.928255 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:55.928222    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:55.928735 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:55.928222    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:55.928735 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:55.928341    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:55.928735 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:55.928387    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:56.084780 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:56.084750    2572 generic.go:356] "Generic (PLEG): container finished" podID="8a690986-5200-4f8f-803d-1db7de256ddb" containerID="0079e5b6881e17115e7d085f8996e9e65d09957f103219e4499b7b4b152d7a0c" exitCode=0
Jul 09 18:04:56.084941 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:56.084834    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9r9n" event={"ID":"8a690986-5200-4f8f-803d-1db7de256ddb","Type":"ContainerDied","Data":"0079e5b6881e17115e7d085f8996e9e65d09957f103219e4499b7b4b152d7a0c"}
Jul 09 18:04:56.101796 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:56.101761    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn" podUID="a6c1dc49-8503-40c7-b286-0d1b9706aec6" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 09 18:04:56.465950 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:56.465920    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:56.466112 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:56.465951    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:56.466112 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:56.466064    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:56.466112 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:56.466067    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:56.466112 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:56.466108    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret podName:f0f42946-27c2-47be-b09b-cefcd2f8ce8e nodeName:}" failed. No retries permitted until 2026-07-09 18:05:28.466096324 +0000 UTC m=+65.304439286 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret") pod "global-pull-secret-syncer-j6pks" (UID: "f0f42946-27c2-47be-b09b-cefcd2f8ce8e") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:56.466270 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:56.466122    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs podName:8695fc30-cdfd-4a63-bf30-33cf14224955 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:28.466115241 +0000 UTC m=+65.304458205 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs") pod "network-metrics-daemon-pw25w" (UID: "8695fc30-cdfd-4a63-bf30-33cf14224955") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:56.567025 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:56.566997    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5jk\" (UniqueName: \"kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk\") pod \"network-check-target-c9jnd\" (UID: \"d50047de-b0e9-4a3c-99ad-b4566e13172c\") " pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:56.567160 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:56.567104    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:56.567160 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:56.567118    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:56.567160 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:56.567126    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cf5jk for pod openshift-network-diagnostics/network-check-target-c9jnd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:56.567269 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:56.567165    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk podName:d50047de-b0e9-4a3c-99ad-b4566e13172c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:28.567154496 +0000 UTC m=+65.405497458 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf5jk" (UniqueName: "kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk") pod "network-check-target-c9jnd" (UID: "d50047de-b0e9-4a3c-99ad-b4566e13172c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:56.928578 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:56.928493    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:56.928918 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:56.928596    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:57.928227 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:57.928202    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:57.928227 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:57.928223    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:57.928461 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:57.928318    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:57.928461 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:57.928445    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:04:58.928021 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:58.927990    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:04:58.928448 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:58.928103    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:04:59.092213 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:59.092184    2572 generic.go:356] "Generic (PLEG): container finished" podID="8a690986-5200-4f8f-803d-1db7de256ddb" containerID="e6c83972751569e2c296dd7629c8a00064a3194d88b9605440d0c4825ffa2ac4" exitCode=0
Jul 09 18:04:59.092359 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:59.092219    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9r9n" event={"ID":"8a690986-5200-4f8f-803d-1db7de256ddb","Type":"ContainerDied","Data":"e6c83972751569e2c296dd7629c8a00064a3194d88b9605440d0c4825ffa2ac4"}
Jul 09 18:04:59.928676 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:59.928643    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:04:59.928676 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:04:59.928680    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:04:59.929135 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:59.928766    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:04:59.929135 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:04:59.928886    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:05:00.928342 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:00.928311    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:05:00.928509 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:00.928440    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:05:01.928719 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:01.928676    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:05:01.929156 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:01.928815    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:05:01.929156 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:01.928865    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:05:01.929156 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:01.928964    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:05:02.927992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:02.927955    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:05:02.928177 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:02.928065    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:05:03.928408 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:03.928374    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:05:03.928857 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:03.928467    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:05:03.928857 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:03.928548    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:05:03.928857 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:03.928641    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:05:04.104039 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:04.104009    2572 generic.go:356] "Generic (PLEG): container finished" podID="8a690986-5200-4f8f-803d-1db7de256ddb" containerID="d5959d4ad843200a4a7c4373d3d5191ddc68152750f0e7b2e1a7e106bd114619" exitCode=0
Jul 09 18:05:04.104213 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:04.104072    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9r9n" event={"ID":"8a690986-5200-4f8f-803d-1db7de256ddb","Type":"ContainerDied","Data":"d5959d4ad843200a4a7c4373d3d5191ddc68152750f0e7b2e1a7e106bd114619"}
Jul 09 18:05:04.928046 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:04.928015    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:05:04.928221 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:04.928123    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:05:05.109132 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:05.109106    2572 generic.go:356] "Generic (PLEG): container finished" podID="8a690986-5200-4f8f-803d-1db7de256ddb" containerID="f0a913126eaf5f498cd1ed2ae78dfd2d9d61615b8c005330618992b2a1a2b6b0" exitCode=0
Jul 09 18:05:05.109606 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:05.109145    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9r9n" event={"ID":"8a690986-5200-4f8f-803d-1db7de256ddb","Type":"ContainerDied","Data":"f0a913126eaf5f498cd1ed2ae78dfd2d9d61615b8c005330618992b2a1a2b6b0"}
Jul 09 18:05:05.928747 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:05.928579    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:05:05.928872 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:05.928609    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:05:05.928931 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:05.928871    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:05:05.928984 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:05.928928    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:05:06.017567 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:06.017523    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c9jnd"]
Jul 09 18:05:06.028499 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:06.028471    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j6pks"]
Jul 09 18:05:06.028648 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:06.028594    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:05:06.028721 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:06.028698    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:05:06.031517 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:06.031496    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pw25w"]
Jul 09 18:05:06.116288 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:06.116213    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9r9n" event={"ID":"8a690986-5200-4f8f-803d-1db7de256ddb","Type":"ContainerStarted","Data":"8db2e1a10a505e1e7a12f143c5f1080d3c6da52d2264f987fa8e5a568f13354e"}
Jul 09 18:05:06.116288 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:06.116236    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:05:06.116288 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:06.116254    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:05:06.116744 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:06.116361    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:05:06.116744 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:06.116504    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:05:06.155969 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:06.155931    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t9r9n" podStartSLOduration=9.132239718 podStartE2EDuration="42.155919077s" totalImagesPullingTime="33.023679359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:06.15575457 +0000 UTC m=+42.994097565" watchObservedRunningTime="2026-07-09 18:05:06.155919077 +0000 UTC m=+42.994262061"
Jul 09 18:05:07.928078 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:07.927989    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:05:07.928078 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:07.928001    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:05:07.928707 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:07.928083    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:05:07.928707 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:07.928168    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:05:07.928707 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:07.928001    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:05:07.928707 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:07.928246    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:05:09.927985 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:09.927955    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:05:09.928634 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:09.928057    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pw25w" podUID="8695fc30-cdfd-4a63-bf30-33cf14224955"
Jul 09 18:05:09.928634 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:09.928095    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:05:09.928634 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:09.928189    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:05:09.928634 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:09.928194    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9jnd" podUID="d50047de-b0e9-4a3c-99ad-b4566e13172c"
Jul 09 18:05:09.928634 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:09.928244    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6pks" podUID="f0f42946-27c2-47be-b09b-cefcd2f8ce8e"
Jul 09 18:05:10.494437 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.494414    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-107.ec2.internal" event="NodeReady"
Jul 09 18:05:10.494595 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.494506    2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:05:10.573755 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.573681    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6m2nr"]
Jul 09 18:05:10.595571 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.595539    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2s9k4"]
Jul 09 18:05:10.595676 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.595639    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6m2nr"
Jul 09 18:05:10.599632 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.599609    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:05:10.599762 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.599685    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2bhtg\""
Jul 09 18:05:10.599762 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.599687    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:05:10.604243 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.604224    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:05:10.617770 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.617749    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6m2nr"]
Jul 09 18:05:10.617855 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.617775    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2s9k4"]
Jul 09 18:05:10.617891 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.617855    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.623827 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.623811    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:05:10.623931 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.623862    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vl9c6\""
Jul 09 18:05:10.624041 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.624029    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:05:10.666464 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.666440    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lzb6m"]
Jul 09 18:05:10.669114 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.669097    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3649103d-34a2-489c-9247-c7a5e928d5d1-cert\") pod \"ingress-canary-6m2nr\" (UID: \"3649103d-34a2-489c-9247-c7a5e928d5d1\") " pod="openshift-ingress-canary/ingress-canary-6m2nr"
Jul 09 18:05:10.669186 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.669145    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2441d9d-b403-4087-842f-efc031230117-config-volume\") pod \"dns-default-2s9k4\" (UID: \"f2441d9d-b403-4087-842f-efc031230117\") " pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.669246 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.669205    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2441d9d-b403-4087-842f-efc031230117-tmp-dir\") pod \"dns-default-2s9k4\" (UID: \"f2441d9d-b403-4087-842f-efc031230117\") " pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.669299 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.669250    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7rqh\" (UniqueName: \"kubernetes.io/projected/3649103d-34a2-489c-9247-c7a5e928d5d1-kube-api-access-j7rqh\") pod \"ingress-canary-6m2nr\" (UID: \"3649103d-34a2-489c-9247-c7a5e928d5d1\") " pod="openshift-ingress-canary/ingress-canary-6m2nr"
Jul 09 18:05:10.669299 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.669286    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6nx\" (UniqueName: \"kubernetes.io/projected/f2441d9d-b403-4087-842f-efc031230117-kube-api-access-zd6nx\") pod \"dns-default-2s9k4\" (UID: \"f2441d9d-b403-4087-842f-efc031230117\") " pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.669396 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.669324    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2441d9d-b403-4087-842f-efc031230117-metrics-tls\") pod \"dns-default-2s9k4\" (UID: \"f2441d9d-b403-4087-842f-efc031230117\") " pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.683672 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.683655    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.690114 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.690034    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lzb6m"]
Jul 09 18:05:10.694927 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.694905    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-htlzv\""
Jul 09 18:05:10.695140 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.695123    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:05:10.695491 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.695475    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:05:10.695528 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.695485    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:05:10.695586 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.695483    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:05:10.770127 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770104    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2441d9d-b403-4087-842f-efc031230117-config-volume\") pod \"dns-default-2s9k4\" (UID: \"f2441d9d-b403-4087-842f-efc031230117\") " pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.770267 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770135    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2441d9d-b403-4087-842f-efc031230117-tmp-dir\") pod \"dns-default-2s9k4\" (UID: \"f2441d9d-b403-4087-842f-efc031230117\") " pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.770267 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770154    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eda6d174-b6cd-4142-a416-ef8d1747ebf5-data-volume\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.770267 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770173    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7rqh\" (UniqueName: \"kubernetes.io/projected/3649103d-34a2-489c-9247-c7a5e928d5d1-kube-api-access-j7rqh\") pod \"ingress-canary-6m2nr\" (UID: \"3649103d-34a2-489c-9247-c7a5e928d5d1\") " pod="openshift-ingress-canary/ingress-canary-6m2nr"
Jul 09 18:05:10.770267 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770191    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eda6d174-b6cd-4142-a416-ef8d1747ebf5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.770267 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770222    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6nx\" (UniqueName: \"kubernetes.io/projected/f2441d9d-b403-4087-842f-efc031230117-kube-api-access-zd6nx\") pod \"dns-default-2s9k4\" (UID: \"f2441d9d-b403-4087-842f-efc031230117\") " pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.770520 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770342    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2441d9d-b403-4087-842f-efc031230117-metrics-tls\") pod \"dns-default-2s9k4\" (UID: \"f2441d9d-b403-4087-842f-efc031230117\") " pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.770520 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770396    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eda6d174-b6cd-4142-a416-ef8d1747ebf5-crio-socket\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.770520 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770425    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eda6d174-b6cd-4142-a416-ef8d1747ebf5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.770520 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770453    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkgw7\" (UniqueName: \"kubernetes.io/projected/eda6d174-b6cd-4142-a416-ef8d1747ebf5-kube-api-access-pkgw7\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.770742 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770502    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3649103d-34a2-489c-9247-c7a5e928d5d1-cert\") pod \"ingress-canary-6m2nr\" (UID: \"3649103d-34a2-489c-9247-c7a5e928d5d1\") " pod="openshift-ingress-canary/ingress-canary-6m2nr"
Jul 09 18:05:10.770742 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770545    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2441d9d-b403-4087-842f-efc031230117-tmp-dir\") pod \"dns-default-2s9k4\" (UID: \"f2441d9d-b403-4087-842f-efc031230117\") " pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.770742 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.770714    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2441d9d-b403-4087-842f-efc031230117-config-volume\") pod \"dns-default-2s9k4\" (UID: \"f2441d9d-b403-4087-842f-efc031230117\") " pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.774140 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.774125    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2441d9d-b403-4087-842f-efc031230117-metrics-tls\") pod \"dns-default-2s9k4\" (UID: \"f2441d9d-b403-4087-842f-efc031230117\") " pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.774215 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.774199    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3649103d-34a2-489c-9247-c7a5e928d5d1-cert\") pod \"ingress-canary-6m2nr\" (UID: \"3649103d-34a2-489c-9247-c7a5e928d5d1\") " pod="openshift-ingress-canary/ingress-canary-6m2nr"
Jul 09 18:05:10.781039 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.781021    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7rqh\" (UniqueName: \"kubernetes.io/projected/3649103d-34a2-489c-9247-c7a5e928d5d1-kube-api-access-j7rqh\") pod \"ingress-canary-6m2nr\" (UID: \"3649103d-34a2-489c-9247-c7a5e928d5d1\") " pod="openshift-ingress-canary/ingress-canary-6m2nr"
Jul 09 18:05:10.783823 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.783790    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6nx\" (UniqueName: \"kubernetes.io/projected/f2441d9d-b403-4087-842f-efc031230117-kube-api-access-zd6nx\") pod \"dns-default-2s9k4\" (UID: \"f2441d9d-b403-4087-842f-efc031230117\") " pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.871448 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.871395    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eda6d174-b6cd-4142-a416-ef8d1747ebf5-data-volume\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.871448 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.871423    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eda6d174-b6cd-4142-a416-ef8d1747ebf5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.871635 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.871460    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eda6d174-b6cd-4142-a416-ef8d1747ebf5-crio-socket\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.871635 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.871478    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eda6d174-b6cd-4142-a416-ef8d1747ebf5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.871635 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.871497    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkgw7\" (UniqueName: \"kubernetes.io/projected/eda6d174-b6cd-4142-a416-ef8d1747ebf5-kube-api-access-pkgw7\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.871785 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.871673    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eda6d174-b6cd-4142-a416-ef8d1747ebf5-crio-socket\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.871843 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.871779    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eda6d174-b6cd-4142-a416-ef8d1747ebf5-data-volume\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.872036 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.872019    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eda6d174-b6cd-4142-a416-ef8d1747ebf5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.873499 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.873481    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eda6d174-b6cd-4142-a416-ef8d1747ebf5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.882440 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.882420    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkgw7\" (UniqueName: \"kubernetes.io/projected/eda6d174-b6cd-4142-a416-ef8d1747ebf5-kube-api-access-pkgw7\") pod \"insights-runtime-extractor-lzb6m\" (UID: \"eda6d174-b6cd-4142-a416-ef8d1747ebf5\") " pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:10.905378 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.905360    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6m2nr"
Jul 09 18:05:10.925533 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.925514    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:10.992032 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:10.992001    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lzb6m"
Jul 09 18:05:11.126406 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:11.126133    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2s9k4"]
Jul 09 18:05:11.129980 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:11.129950    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2441d9d_b403_4087_842f_efc031230117.slice/crio-6cf78f31d5f8c0db6a2a5766ee967ed94cd8abfb8ff7c43a92266ff1f58ac603 WatchSource:0}: Error finding container 6cf78f31d5f8c0db6a2a5766ee967ed94cd8abfb8ff7c43a92266ff1f58ac603: Status 404 returned error can't find the container with id 6cf78f31d5f8c0db6a2a5766ee967ed94cd8abfb8ff7c43a92266ff1f58ac603
Jul 09 18:05:11.138847 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:11.138824    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6m2nr"]
Jul 09 18:05:11.141295 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:11.141268    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3649103d_34a2_489c_9247_c7a5e928d5d1.slice/crio-0f0482144b5c4aec9c24e2e64958651f656d756b4d81086cd42330a6a9fbe463 WatchSource:0}: Error finding container 0f0482144b5c4aec9c24e2e64958651f656d756b4d81086cd42330a6a9fbe463: Status 404 returned error can't find the container with id 0f0482144b5c4aec9c24e2e64958651f656d756b4d81086cd42330a6a9fbe463
Jul 09 18:05:11.174471 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:11.174448    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lzb6m"]
Jul 09 18:05:11.179123 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:11.179103    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda6d174_b6cd_4142_a416_ef8d1747ebf5.slice/crio-936c5d0891c473fc7bbb56d503693418661ae34b8fc8d2d201cb831a7bb7b726 WatchSource:0}: Error finding container 936c5d0891c473fc7bbb56d503693418661ae34b8fc8d2d201cb831a7bb7b726: Status 404 returned error can't find the container with id 936c5d0891c473fc7bbb56d503693418661ae34b8fc8d2d201cb831a7bb7b726
Jul 09 18:05:11.927688 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:11.927655    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:05:11.928095 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:11.928080    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:05:11.928870 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:11.928492    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:05:11.933429 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:11.931191    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:11.933429 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:11.933040    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-skcpw\""
Jul 09 18:05:11.935668 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:11.934669    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.935668 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:11.934923    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mw97p\""
Jul 09 18:05:11.935668 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:11.935173    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:11.935668 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:11.935446    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:12.134347 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:12.134306    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzb6m" event={"ID":"eda6d174-b6cd-4142-a416-ef8d1747ebf5","Type":"ContainerStarted","Data":"936c5d0891c473fc7bbb56d503693418661ae34b8fc8d2d201cb831a7bb7b726"}
Jul 09 18:05:12.136089 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:12.136020    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2s9k4" event={"ID":"f2441d9d-b403-4087-842f-efc031230117","Type":"ContainerStarted","Data":"6cf78f31d5f8c0db6a2a5766ee967ed94cd8abfb8ff7c43a92266ff1f58ac603"}
Jul 09 18:05:12.137878 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:12.137816    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6m2nr" event={"ID":"3649103d-34a2-489c-9247-c7a5e928d5d1","Type":"ContainerStarted","Data":"0f0482144b5c4aec9c24e2e64958651f656d756b4d81086cd42330a6a9fbe463"}
Jul 09 18:05:15.042098 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.042051    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ftgm4"]
Jul 09 18:05:15.127279 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.127249    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ftgm4"]
Jul 09 18:05:15.127447 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.127282    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.135054 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.134877    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 09 18:05:15.135054 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.134905    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:05:15.135054 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.134944    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 09 18:05:15.135054 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.134947    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:05:15.135330 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.135202    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:05:15.135330 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.135265    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dfvvs\""
Jul 09 18:05:15.204326 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.204292    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acc69e44-4fc3-4a24-9a16-fa2e27ab582b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ftgm4\" (UID: \"acc69e44-4fc3-4a24-9a16-fa2e27ab582b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.204496 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.204337    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acc69e44-4fc3-4a24-9a16-fa2e27ab582b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ftgm4\" (UID: \"acc69e44-4fc3-4a24-9a16-fa2e27ab582b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.204496 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.204380    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acc69e44-4fc3-4a24-9a16-fa2e27ab582b-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ftgm4\" (UID: \"acc69e44-4fc3-4a24-9a16-fa2e27ab582b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.204496 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.204413    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nvns\" (UniqueName: \"kubernetes.io/projected/acc69e44-4fc3-4a24-9a16-fa2e27ab582b-kube-api-access-2nvns\") pod \"prometheus-operator-64785b6675-ftgm4\" (UID: \"acc69e44-4fc3-4a24-9a16-fa2e27ab582b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.305640 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.305565    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acc69e44-4fc3-4a24-9a16-fa2e27ab582b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ftgm4\" (UID: \"acc69e44-4fc3-4a24-9a16-fa2e27ab582b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.305640 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.305600    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acc69e44-4fc3-4a24-9a16-fa2e27ab582b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ftgm4\" (UID: \"acc69e44-4fc3-4a24-9a16-fa2e27ab582b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.305640 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.305625    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acc69e44-4fc3-4a24-9a16-fa2e27ab582b-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ftgm4\" (UID: \"acc69e44-4fc3-4a24-9a16-fa2e27ab582b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.305878 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.305653    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nvns\" (UniqueName: \"kubernetes.io/projected/acc69e44-4fc3-4a24-9a16-fa2e27ab582b-kube-api-access-2nvns\") pod \"prometheus-operator-64785b6675-ftgm4\" (UID: \"acc69e44-4fc3-4a24-9a16-fa2e27ab582b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.306401 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.306373    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acc69e44-4fc3-4a24-9a16-fa2e27ab582b-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ftgm4\" (UID: \"acc69e44-4fc3-4a24-9a16-fa2e27ab582b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.309070 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.309043    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acc69e44-4fc3-4a24-9a16-fa2e27ab582b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ftgm4\" (UID: \"acc69e44-4fc3-4a24-9a16-fa2e27ab582b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.309186 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.309105    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acc69e44-4fc3-4a24-9a16-fa2e27ab582b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ftgm4\" (UID: \"acc69e44-4fc3-4a24-9a16-fa2e27ab582b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.342465 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.342440    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-df2bx"]
Jul 09 18:05:15.371208 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.371177    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nvns\" (UniqueName: \"kubernetes.io/projected/acc69e44-4fc3-4a24-9a16-fa2e27ab582b-kube-api-access-2nvns\") pod \"prometheus-operator-64785b6675-ftgm4\" (UID: \"acc69e44-4fc3-4a24-9a16-fa2e27ab582b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.400753 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.400729    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-df2bx"]
Jul 09 18:05:15.400889 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.400864    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-df2bx"
Jul 09 18:05:15.405280 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.405258    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 18:05:15.405385 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.405353    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5g9bx\""
Jul 09 18:05:15.407916 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.407899    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 18:05:15.437671 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.437634    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4"
Jul 09 18:05:15.507895 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.507870    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwlfc\" (UniqueName: \"kubernetes.io/projected/da80f5f7-dfed-474e-96e7-c60902e0f08a-kube-api-access-gwlfc\") pod \"downloads-68c9cbb497-df2bx\" (UID: \"da80f5f7-dfed-474e-96e7-c60902e0f08a\") " pod="openshift-console/downloads-68c9cbb497-df2bx"
Jul 09 18:05:15.608825 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.608801    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwlfc\" (UniqueName: \"kubernetes.io/projected/da80f5f7-dfed-474e-96e7-c60902e0f08a-kube-api-access-gwlfc\") pod \"downloads-68c9cbb497-df2bx\" (UID: \"da80f5f7-dfed-474e-96e7-c60902e0f08a\") " pod="openshift-console/downloads-68c9cbb497-df2bx"
Jul 09 18:05:15.648090 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.648072    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwlfc\" (UniqueName: \"kubernetes.io/projected/da80f5f7-dfed-474e-96e7-c60902e0f08a-kube-api-access-gwlfc\") pod \"downloads-68c9cbb497-df2bx\" (UID: \"da80f5f7-dfed-474e-96e7-c60902e0f08a\") " pod="openshift-console/downloads-68c9cbb497-df2bx"
Jul 09 18:05:15.711316 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:15.711293    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-df2bx"
Jul 09 18:05:17.116965 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:17.116786    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-df2bx"]
Jul 09 18:05:17.119817 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:17.119781    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda80f5f7_dfed_474e_96e7_c60902e0f08a.slice/crio-69a41132d204095c2306e40f84943baeffb110a19c5c602b0cc5fd3d8ddbb7de WatchSource:0}: Error finding container 69a41132d204095c2306e40f84943baeffb110a19c5c602b0cc5fd3d8ddbb7de: Status 404 returned error can't find the container with id 69a41132d204095c2306e40f84943baeffb110a19c5c602b0cc5fd3d8ddbb7de
Jul 09 18:05:17.122576 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:17.122161    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ftgm4"]
Jul 09 18:05:17.126215 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:17.126195    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc69e44_4fc3_4a24_9a16_fa2e27ab582b.slice/crio-ff188413fe1e1772dd7b6de51c04a6e467d0dd42e7202ea4dbfc752e75b48790 WatchSource:0}: Error finding container ff188413fe1e1772dd7b6de51c04a6e467d0dd42e7202ea4dbfc752e75b48790: Status 404 returned error can't find the container with id ff188413fe1e1772dd7b6de51c04a6e467d0dd42e7202ea4dbfc752e75b48790
Jul 09 18:05:17.153833 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:17.153800    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2s9k4" event={"ID":"f2441d9d-b403-4087-842f-efc031230117","Type":"ContainerStarted","Data":"55912cc19c3ed820bee1d20882d32dc4fa549df55605454a62c614cb09f7f570"}
Jul 09 18:05:17.155432 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:17.155405    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6m2nr" event={"ID":"3649103d-34a2-489c-9247-c7a5e928d5d1","Type":"ContainerStarted","Data":"f1bdcb37cd08a75f54af337497a8777b9c96f4b30ca77854f9d3ab7472de5377"}
Jul 09 18:05:17.156772 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:17.156747    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-df2bx" event={"ID":"da80f5f7-dfed-474e-96e7-c60902e0f08a","Type":"ContainerStarted","Data":"69a41132d204095c2306e40f84943baeffb110a19c5c602b0cc5fd3d8ddbb7de"}
Jul 09 18:05:17.158119 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:17.157948    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4" event={"ID":"acc69e44-4fc3-4a24-9a16-fa2e27ab582b","Type":"ContainerStarted","Data":"ff188413fe1e1772dd7b6de51c04a6e467d0dd42e7202ea4dbfc752e75b48790"}
Jul 09 18:05:17.184410 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:17.184365    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6m2nr" podStartSLOduration=1.366502043 podStartE2EDuration="7.184349059s" totalImagesPullingTime="5.817847016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:17.18372384 +0000 UTC m=+54.022066824" watchObservedRunningTime="2026-07-09 18:05:17.184349059 +0000 UTC m=+54.022692045"
Jul 09 18:05:18.162530 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:18.162487    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzb6m" event={"ID":"eda6d174-b6cd-4142-a416-ef8d1747ebf5","Type":"ContainerStarted","Data":"cd373f42103be1efe525ee1991b1d5001aee224605d841998f556f64945ff3fe"}
Jul 09 18:05:18.164527 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:18.164491    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2s9k4" event={"ID":"f2441d9d-b403-4087-842f-efc031230117","Type":"ContainerStarted","Data":"e3f2fac16a0a733c317377ac1211092ea47c547740545da050a19328d6d03ceb"}
Jul 09 18:05:18.191302 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:18.191258    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2s9k4" podStartSLOduration=2.362321919 podStartE2EDuration="8.191243905s" totalImagesPullingTime="5.828921986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:18.191060192 +0000 UTC m=+55.029403192" watchObservedRunningTime="2026-07-09 18:05:18.191243905 +0000 UTC m=+55.029586888"
Jul 09 18:05:19.169648 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:19.169451    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzb6m" event={"ID":"eda6d174-b6cd-4142-a416-ef8d1747ebf5","Type":"ContainerStarted","Data":"3ec1c0399b6af7ee089821108d05a7c48c0b6a38889c99526e71071fb3aa5efd"}
Jul 09 18:05:19.171011 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:19.170988    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4" event={"ID":"acc69e44-4fc3-4a24-9a16-fa2e27ab582b","Type":"ContainerStarted","Data":"e055f8dd7b801b8e1a884845d1fb890d84fe086b4deb6d0041663a99660771c1"}
Jul 09 18:05:19.171106 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:19.171032    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4" event={"ID":"acc69e44-4fc3-4a24-9a16-fa2e27ab582b","Type":"ContainerStarted","Data":"8eae937071804dd9688b9fa9b7c19beb0a7c67ed85dfa595f6739ce8521a8708"}
Jul 09 18:05:19.171106 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:19.171081    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:19.195965 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:19.195922    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-ftgm4" podStartSLOduration=2.7592500810000002 podStartE2EDuration="4.195906261s" totalImagesPullingTime="1.43665618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:19.195193292 +0000 UTC m=+56.033536286" watchObservedRunningTime="2026-07-09 18:05:19.195906261 +0000 UTC m=+56.034249245"
Jul 09 18:05:21.279771 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.279734    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-598db9cd7b-sfsh9"]
Jul 09 18:05:21.283005 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.282975    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.287284 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.287257    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 18:05:21.288643 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.288425    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 18:05:21.288643 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.288456    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 18:05:21.288827 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.288727    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6txwz\""
Jul 09 18:05:21.288827 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.288774    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 18:05:21.288999 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.288957    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 18:05:21.300574 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.300530    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-598db9cd7b-sfsh9"]
Jul 09 18:05:21.352078 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.352052    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/094645b5-358b-4428-8564-296efebbbf21-console-oauth-config\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.352205 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.352094    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-oauth-serving-cert\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.352205 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.352117    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-service-ca\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.352325 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.352207    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48lks\" (UniqueName: \"kubernetes.io/projected/094645b5-358b-4428-8564-296efebbbf21-kube-api-access-48lks\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.352325 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.352259    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-console-config\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.352325 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.352297    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/094645b5-358b-4428-8564-296efebbbf21-console-serving-cert\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.453479 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.453446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/094645b5-358b-4428-8564-296efebbbf21-console-oauth-config\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.453647 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.453486    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-oauth-serving-cert\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.453647 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.453517    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-service-ca\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.453647 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.453584    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48lks\" (UniqueName: \"kubernetes.io/projected/094645b5-358b-4428-8564-296efebbbf21-kube-api-access-48lks\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.453647 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.453627    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-console-config\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.453860 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.453659    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/094645b5-358b-4428-8564-296efebbbf21-console-serving-cert\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.454784 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.454733    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-oauth-serving-cert\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.454893 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.454800    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-service-ca\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.455005 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.454983    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-console-config\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.458835 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.458624    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/094645b5-358b-4428-8564-296efebbbf21-console-oauth-config\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.458835 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.458680    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/094645b5-358b-4428-8564-296efebbbf21-console-serving-cert\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.470841 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.470818    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48lks\" (UniqueName: \"kubernetes.io/projected/094645b5-358b-4428-8564-296efebbbf21-kube-api-access-48lks\") pod \"console-598db9cd7b-sfsh9\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") " pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.594781 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.594628    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:21.718513 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.718482    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"]
Jul 09 18:05:21.722602 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.722576    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.726768 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.726719    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 09 18:05:21.726906 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.726885    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 09 18:05:21.726980 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.726887    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-w27g5\""
Jul 09 18:05:21.755979 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.755945    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8603f65d-3b7e-4e30-92fe-70e2df00c022-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-kvvrm\" (UID: \"8603f65d-3b7e-4e30-92fe-70e2df00c022\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.756091 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.755995    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8603f65d-3b7e-4e30-92fe-70e2df00c022-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-kvvrm\" (UID: \"8603f65d-3b7e-4e30-92fe-70e2df00c022\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.756159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.756088    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8603f65d-3b7e-4e30-92fe-70e2df00c022-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-kvvrm\" (UID: \"8603f65d-3b7e-4e30-92fe-70e2df00c022\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.756159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.756118    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc8b6\" (UniqueName: \"kubernetes.io/projected/8603f65d-3b7e-4e30-92fe-70e2df00c022-kube-api-access-qc8b6\") pod \"openshift-state-metrics-84bd68b896-kvvrm\" (UID: \"8603f65d-3b7e-4e30-92fe-70e2df00c022\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.762082 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.762058    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"]
Jul 09 18:05:21.762870 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.762844    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-blzln"]
Jul 09 18:05:21.765765 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.765748    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.768990 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.768962    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:05:21.769088 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.768963    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:05:21.769088 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.769059    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mtfc8\""
Jul 09 18:05:21.769088 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.769079    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:05:21.785479 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.785458    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-98lvl"]
Jul 09 18:05:21.788271 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.788254    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.791171 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.791155    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 09 18:05:21.792713 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.792696    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xrvt6\""
Jul 09 18:05:21.793102 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.793069    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 09 18:05:21.793689 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.793670    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 09 18:05:21.817996 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.817957    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-98lvl"]
Jul 09 18:05:21.856482 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856406    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/98cd0934-0a28-43e2-8fde-32a6dbea4697-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.856482 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856443    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac47df41-c136-4bd8-a0ca-754f5809be16-metrics-client-ca\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.856482 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856478    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8603f65d-3b7e-4e30-92fe-70e2df00c022-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-kvvrm\" (UID: \"8603f65d-3b7e-4e30-92fe-70e2df00c022\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.856720 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856590    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9wjl\" (UniqueName: \"kubernetes.io/projected/98cd0934-0a28-43e2-8fde-32a6dbea4697-kube-api-access-p9wjl\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.856720 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856630    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-accelerators-collector-config\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.856720 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856662    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8603f65d-3b7e-4e30-92fe-70e2df00c022-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-kvvrm\" (UID: \"8603f65d-3b7e-4e30-92fe-70e2df00c022\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.856720 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856687    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/98cd0934-0a28-43e2-8fde-32a6dbea4697-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.856720 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856710    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-tls\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.856913 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856733    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.856913 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856758    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac47df41-c136-4bd8-a0ca-754f5809be16-root\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.856913 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856796    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98cd0934-0a28-43e2-8fde-32a6dbea4697-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.856913 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856826    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-wtmp\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.857105 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856906    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc8b6\" (UniqueName: \"kubernetes.io/projected/8603f65d-3b7e-4e30-92fe-70e2df00c022-kube-api-access-qc8b6\") pod \"openshift-state-metrics-84bd68b896-kvvrm\" (UID: \"8603f65d-3b7e-4e30-92fe-70e2df00c022\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.857105 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856943    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac47df41-c136-4bd8-a0ca-754f5809be16-sys\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.857105 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.856971    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-textfile\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.857105 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.857001    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4wnx\" (UniqueName: \"kubernetes.io/projected/ac47df41-c136-4bd8-a0ca-754f5809be16-kube-api-access-d4wnx\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.857105 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.857060    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8603f65d-3b7e-4e30-92fe-70e2df00c022-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-kvvrm\" (UID: \"8603f65d-3b7e-4e30-92fe-70e2df00c022\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.857105 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.857097    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/98cd0934-0a28-43e2-8fde-32a6dbea4697-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.857393 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.857132    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98cd0934-0a28-43e2-8fde-32a6dbea4697-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.857726 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.857703    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8603f65d-3b7e-4e30-92fe-70e2df00c022-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-kvvrm\" (UID: \"8603f65d-3b7e-4e30-92fe-70e2df00c022\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.859545 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.859502    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8603f65d-3b7e-4e30-92fe-70e2df00c022-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-kvvrm\" (UID: \"8603f65d-3b7e-4e30-92fe-70e2df00c022\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.860092 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.860053    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8603f65d-3b7e-4e30-92fe-70e2df00c022-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-kvvrm\" (UID: \"8603f65d-3b7e-4e30-92fe-70e2df00c022\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.880978 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.880952    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc8b6\" (UniqueName: \"kubernetes.io/projected/8603f65d-3b7e-4e30-92fe-70e2df00c022-kube-api-access-qc8b6\") pod \"openshift-state-metrics-84bd68b896-kvvrm\" (UID: \"8603f65d-3b7e-4e30-92fe-70e2df00c022\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:21.958075 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958050    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/98cd0934-0a28-43e2-8fde-32a6dbea4697-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.958231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958085    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98cd0934-0a28-43e2-8fde-32a6dbea4697-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.958231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958131    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/98cd0934-0a28-43e2-8fde-32a6dbea4697-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.958231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958148    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac47df41-c136-4bd8-a0ca-754f5809be16-metrics-client-ca\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.958414 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958300    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9wjl\" (UniqueName: \"kubernetes.io/projected/98cd0934-0a28-43e2-8fde-32a6dbea4697-kube-api-access-p9wjl\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.958414 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-accelerators-collector-config\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.958414 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958366    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/98cd0934-0a28-43e2-8fde-32a6dbea4697-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.958414 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958393    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-tls\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.958642 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958416    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.958642 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958441    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac47df41-c136-4bd8-a0ca-754f5809be16-root\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.958642 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958462    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/98cd0934-0a28-43e2-8fde-32a6dbea4697-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.958642 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958483    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98cd0934-0a28-43e2-8fde-32a6dbea4697-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.958642 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958514    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-wtmp\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.958642 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958543    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac47df41-c136-4bd8-a0ca-754f5809be16-sys\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.958642 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958589    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-textfile\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.958642 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958620    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wnx\" (UniqueName: \"kubernetes.io/projected/ac47df41-c136-4bd8-a0ca-754f5809be16-kube-api-access-d4wnx\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.959259 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.958748    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac47df41-c136-4bd8-a0ca-754f5809be16-metrics-client-ca\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.959259 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.959017    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/98cd0934-0a28-43e2-8fde-32a6dbea4697-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.959259 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.959050    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac47df41-c136-4bd8-a0ca-754f5809be16-root\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.959259 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.959099    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-wtmp\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.959259 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.959099    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac47df41-c136-4bd8-a0ca-754f5809be16-sys\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.959481 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.959323    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-textfile\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.959481 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.959359    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-accelerators-collector-config\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.959481 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:21.959396    2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 09 18:05:21.959481 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:05:21.959464    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-tls podName:ac47df41-c136-4bd8-a0ca-754f5809be16 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:22.459445703 +0000 UTC m=+59.297788666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-tls") pod "node-exporter-blzln" (UID: "ac47df41-c136-4bd8-a0ca-754f5809be16") : secret "node-exporter-tls" not found
Jul 09 18:05:21.959818 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.959750    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98cd0934-0a28-43e2-8fde-32a6dbea4697-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.960916 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.960876    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98cd0934-0a28-43e2-8fde-32a6dbea4697-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.962071 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.962050    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.962494 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.962472    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/98cd0934-0a28-43e2-8fde-32a6dbea4697-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:21.990772 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.990732    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4wnx\" (UniqueName: \"kubernetes.io/projected/ac47df41-c136-4bd8-a0ca-754f5809be16-kube-api-access-d4wnx\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:21.993975 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:21.993951    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9wjl\" (UniqueName: \"kubernetes.io/projected/98cd0934-0a28-43e2-8fde-32a6dbea4697-kube-api-access-p9wjl\") pod \"kube-state-metrics-785778ff74-98lvl\" (UID: \"98cd0934-0a28-43e2-8fde-32a6dbea4697\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:22.032784 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:22.032753    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"
Jul 09 18:05:22.098790 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:22.098766    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl"
Jul 09 18:05:22.396651 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:22.396620    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-598db9cd7b-sfsh9"]
Jul 09 18:05:22.400113 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:22.400084    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094645b5_358b_4428_8564_296efebbbf21.slice/crio-b488aeb2d3c9a89a728f9bed2e93dd7ea3e8e424d4f095757718bfb03d8f35ca WatchSource:0}: Error finding container b488aeb2d3c9a89a728f9bed2e93dd7ea3e8e424d4f095757718bfb03d8f35ca: Status 404 returned error can't find the container with id b488aeb2d3c9a89a728f9bed2e93dd7ea3e8e424d4f095757718bfb03d8f35ca
Jul 09 18:05:22.407227 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:22.407204    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm"]
Jul 09 18:05:22.412595 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:22.412549    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8603f65d_3b7e_4e30_92fe_70e2df00c022.slice/crio-797448682fb4b515b2e6624725f9a9779fe2474568db7cddc92cb60cd2b63f50 WatchSource:0}: Error finding container 797448682fb4b515b2e6624725f9a9779fe2474568db7cddc92cb60cd2b63f50: Status 404 returned error can't find the container with id 797448682fb4b515b2e6624725f9a9779fe2474568db7cddc92cb60cd2b63f50
Jul 09 18:05:22.434913 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:22.434886    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-98lvl"]
Jul 09 18:05:22.437462 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:22.437436    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cd0934_0a28_43e2_8fde_32a6dbea4697.slice/crio-baa33a2a11292aab0d0e5d004d6539fe3d6a603e3c3c0c9e3ccc7f88ee0b6d45 WatchSource:0}: Error finding container baa33a2a11292aab0d0e5d004d6539fe3d6a603e3c3c0c9e3ccc7f88ee0b6d45: Status 404 returned error can't find the container with id baa33a2a11292aab0d0e5d004d6539fe3d6a603e3c3c0c9e3ccc7f88ee0b6d45
Jul 09 18:05:22.462668 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:22.462645    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-tls\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:22.465099 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:22.465075    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac47df41-c136-4bd8-a0ca-754f5809be16-node-exporter-tls\") pod \"node-exporter-blzln\" (UID: \"ac47df41-c136-4bd8-a0ca-754f5809be16\") " pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:22.676863 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:22.676795    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-blzln"
Jul 09 18:05:22.687123 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:22.687095    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac47df41_c136_4bd8_a0ca_754f5809be16.slice/crio-6748d096dbdabe2d7c326d02c39b4859b702775fa50b522b10a5a49ae539ed06 WatchSource:0}: Error finding container 6748d096dbdabe2d7c326d02c39b4859b702775fa50b522b10a5a49ae539ed06: Status 404 returned error can't find the container with id 6748d096dbdabe2d7c326d02c39b4859b702775fa50b522b10a5a49ae539ed06
Jul 09 18:05:23.195111 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:23.195033    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzb6m" event={"ID":"eda6d174-b6cd-4142-a416-ef8d1747ebf5","Type":"ContainerStarted","Data":"3907ecf25b7ca597662ab97d1032566964a62d87551a677d62b12f018d8a2b0b"}
Jul 09 18:05:23.197164 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:23.197137    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-598db9cd7b-sfsh9" event={"ID":"094645b5-358b-4428-8564-296efebbbf21","Type":"ContainerStarted","Data":"b488aeb2d3c9a89a728f9bed2e93dd7ea3e8e424d4f095757718bfb03d8f35ca"}
Jul 09 18:05:23.199841 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:23.199715    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm" event={"ID":"8603f65d-3b7e-4e30-92fe-70e2df00c022","Type":"ContainerStarted","Data":"54bbff99aaf5048e7717717c87b36e2f9ae85d15c170c99010e52d83f73849dc"}
Jul 09 18:05:23.199841 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:23.199773    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm" event={"ID":"8603f65d-3b7e-4e30-92fe-70e2df00c022","Type":"ContainerStarted","Data":"7592b99f34b938565a54fead4975beefee400e86c696be823e1d397ecadc754d"}
Jul 09 18:05:23.199841 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:23.199809    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm" event={"ID":"8603f65d-3b7e-4e30-92fe-70e2df00c022","Type":"ContainerStarted","Data":"797448682fb4b515b2e6624725f9a9779fe2474568db7cddc92cb60cd2b63f50"}
Jul 09 18:05:23.201637 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:23.201583    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl" event={"ID":"98cd0934-0a28-43e2-8fde-32a6dbea4697","Type":"ContainerStarted","Data":"baa33a2a11292aab0d0e5d004d6539fe3d6a603e3c3c0c9e3ccc7f88ee0b6d45"}
Jul 09 18:05:23.203507 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:23.203444    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blzln" event={"ID":"ac47df41-c136-4bd8-a0ca-754f5809be16","Type":"ContainerStarted","Data":"6748d096dbdabe2d7c326d02c39b4859b702775fa50b522b10a5a49ae539ed06"}
Jul 09 18:05:23.226692 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:23.226586    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lzb6m" podStartSLOduration=2.386996545 podStartE2EDuration="13.226548795s" totalImagesPullingTime="10.83955225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:23.22502803 +0000 UTC m=+60.063371018" watchObservedRunningTime="2026-07-09 18:05:23.226548795 +0000 UTC m=+60.064891777"
Jul 09 18:05:25.216017 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:25.215938    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm" event={"ID":"8603f65d-3b7e-4e30-92fe-70e2df00c022","Type":"ContainerStarted","Data":"343fafda604521abf9c4c422e8486ac67f08ad3cb8b92e64fe7d5a2bcd8186f0"}
Jul 09 18:05:25.218842 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:25.218813    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl" event={"ID":"98cd0934-0a28-43e2-8fde-32a6dbea4697","Type":"ContainerStarted","Data":"d4c08458b63c5fedf505fd112d67d843ca09a4d7e96481ced58b1bbab42dc094"}
Jul 09 18:05:25.218952 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:25.218870    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl" event={"ID":"98cd0934-0a28-43e2-8fde-32a6dbea4697","Type":"ContainerStarted","Data":"71eead6d5c6a69985dfa179f818d0aa2a14526c0d6d6da49177d8f66667d578e"}
Jul 09 18:05:25.218952 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:25.218897    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl" event={"ID":"98cd0934-0a28-43e2-8fde-32a6dbea4697","Type":"ContainerStarted","Data":"0d4bc6ab0f8f16738e2cd0b6e248db0bca20da236f3d2bf58928bf8c0b7587e1"}
Jul 09 18:05:25.221307 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:25.221280    2572 generic.go:356] "Generic (PLEG): container finished" podID="ac47df41-c136-4bd8-a0ca-754f5809be16" containerID="bec3512488ae34b916ad6838040837036b9ffb09eabd219a100cff8760e24488" exitCode=0
Jul 09 18:05:25.221435 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:25.221373    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blzln" event={"ID":"ac47df41-c136-4bd8-a0ca-754f5809be16","Type":"ContainerDied","Data":"bec3512488ae34b916ad6838040837036b9ffb09eabd219a100cff8760e24488"}
Jul 09 18:05:25.253086 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:25.252933    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-kvvrm" podStartSLOduration=2.378741209 podStartE2EDuration="4.252915548s" totalImagesPullingTime="1.874174339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:25.252878692 +0000 UTC m=+62.091221676" watchObservedRunningTime="2026-07-09 18:05:25.252915548 +0000 UTC m=+62.091258513"
Jul 09 18:05:25.346271 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:25.346129    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-98lvl" podStartSLOduration=2.419116372 podStartE2EDuration="4.346109345s" totalImagesPullingTime="1.926992973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:25.345790645 +0000 UTC m=+62.184133630" watchObservedRunningTime="2026-07-09 18:05:25.346109345 +0000 UTC m=+62.184452310"
Jul 09 18:05:26.097477 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.097451    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmxsn"
Jul 09 18:05:26.194038 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.194005    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"]
Jul 09 18:05:26.196480 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.196403    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"
Jul 09 18:05:26.199573 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.199534    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 09 18:05:26.199701 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.199577    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5lc6c\""
Jul 09 18:05:26.212886 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.212831    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"]
Jul 09 18:05:26.229101 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.229066    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blzln" event={"ID":"ac47df41-c136-4bd8-a0ca-754f5809be16","Type":"ContainerStarted","Data":"26199215f9cadc76b807d5032521fab22e89204004bb0fbcdbdf5e15870ef78e"}
Jul 09 18:05:26.229523 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.229128    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blzln" event={"ID":"ac47df41-c136-4bd8-a0ca-754f5809be16","Type":"ContainerStarted","Data":"874a3d12ec35f10527f714ec311a339b33673087408008202a0ee65038e75ec3"}
Jul 09 18:05:26.257112 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.257061    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-blzln" podStartSLOduration=3.468762904 podStartE2EDuration="5.257038154s" totalImagesPullingTime="1.68138748s" totalInitContainerRuntime="106.88777ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:26.255760679 +0000 UTC m=+63.094103664" watchObservedRunningTime="2026-07-09 18:05:26.257038154 +0000 UTC m=+63.095381139"
Jul 09 18:05:26.297549 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.297482    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5d9c301-4af3-4b86-8c40-86876f728240-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-s77df\" (UID: \"d5d9c301-4af3-4b86-8c40-86876f728240\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"
Jul 09 18:05:26.297719 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.297607    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlj8\" (UniqueName: \"kubernetes.io/projected/d5d9c301-4af3-4b86-8c40-86876f728240-kube-api-access-5hlj8\") pod \"monitoring-plugin-5f68df7f58-s77df\" (UID: \"d5d9c301-4af3-4b86-8c40-86876f728240\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"
Jul 09 18:05:26.399219 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.399134    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5d9c301-4af3-4b86-8c40-86876f728240-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-s77df\" (UID: \"d5d9c301-4af3-4b86-8c40-86876f728240\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"
Jul 09 18:05:26.399219 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.399189    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlj8\" (UniqueName: \"kubernetes.io/projected/d5d9c301-4af3-4b86-8c40-86876f728240-kube-api-access-5hlj8\") pod \"monitoring-plugin-5f68df7f58-s77df\" (UID: \"d5d9c301-4af3-4b86-8c40-86876f728240\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"
Jul 09 18:05:26.405913 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.405881    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5d9c301-4af3-4b86-8c40-86876f728240-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-s77df\" (UID: \"d5d9c301-4af3-4b86-8c40-86876f728240\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"
Jul 09 18:05:26.420867 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.420840    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlj8\" (UniqueName: \"kubernetes.io/projected/d5d9c301-4af3-4b86-8c40-86876f728240-kube-api-access-5hlj8\") pod \"monitoring-plugin-5f68df7f58-s77df\" (UID: \"d5d9c301-4af3-4b86-8c40-86876f728240\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"
Jul 09 18:05:26.508861 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:26.508817    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"
Jul 09 18:05:27.173896 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.173863    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"]
Jul 09 18:05:27.193322 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.193300    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.203741 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.203723    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 09 18:05:27.204097 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.204081    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 09 18:05:27.205095 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.204896    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 09 18:05:27.205095 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.204978    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 09 18:05:27.205251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.205149    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 09 18:05:27.205648 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.205630    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zjr4h\""
Jul 09 18:05:27.219492 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.219470    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 09 18:05:27.232911 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.232833    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"]
Jul 09 18:05:27.239584 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:27.239531    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d9c301_4af3_4b86_8c40_86876f728240.slice/crio-d19da163c5fdc0f578146d1a2ed77034980af5f0539f7c6cfea3e38270f8b60a WatchSource:0}: Error finding container d19da163c5fdc0f578146d1a2ed77034980af5f0539f7c6cfea3e38270f8b60a: Status 404 returned error can't find the container with id d19da163c5fdc0f578146d1a2ed77034980af5f0539f7c6cfea3e38270f8b60a
Jul 09 18:05:27.306336 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.306277    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8054ec99-6079-4530-8de8-5638d480d32c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.306336 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.306323    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52qd6\" (UniqueName: \"kubernetes.io/projected/8054ec99-6079-4530-8de8-5638d480d32c-kube-api-access-52qd6\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.306574 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.306457    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8054ec99-6079-4530-8de8-5638d480d32c-telemeter-client-tls\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.306574 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.306482    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8054ec99-6079-4530-8de8-5638d480d32c-federate-client-tls\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.307147 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.306945    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8054ec99-6079-4530-8de8-5638d480d32c-metrics-client-ca\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.307147 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.306997    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8054ec99-6079-4530-8de8-5638d480d32c-serving-certs-ca-bundle\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.307147 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.307056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8054ec99-6079-4530-8de8-5638d480d32c-secret-telemeter-client\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.307147 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.307093    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8054ec99-6079-4530-8de8-5638d480d32c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.309439 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.309419    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"]
Jul 09 18:05:27.407889 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.407818    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8054ec99-6079-4530-8de8-5638d480d32c-metrics-client-ca\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.407889 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.407860    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8054ec99-6079-4530-8de8-5638d480d32c-serving-certs-ca-bundle\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.408231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.407897    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8054ec99-6079-4530-8de8-5638d480d32c-secret-telemeter-client\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.408231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.408096    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8054ec99-6079-4530-8de8-5638d480d32c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.408231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.408147    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8054ec99-6079-4530-8de8-5638d480d32c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.408231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.408171    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52qd6\" (UniqueName: \"kubernetes.io/projected/8054ec99-6079-4530-8de8-5638d480d32c-kube-api-access-52qd6\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.408424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.408241    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8054ec99-6079-4530-8de8-5638d480d32c-telemeter-client-tls\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.408424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.408266    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8054ec99-6079-4530-8de8-5638d480d32c-federate-client-tls\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.408840 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.408788    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8054ec99-6079-4530-8de8-5638d480d32c-metrics-client-ca\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.408840 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.408813    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8054ec99-6079-4530-8de8-5638d480d32c-serving-certs-ca-bundle\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.409402 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.409355    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8054ec99-6079-4530-8de8-5638d480d32c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.411192 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.411166    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8054ec99-6079-4530-8de8-5638d480d32c-secret-telemeter-client\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.411380 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.411357    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8054ec99-6079-4530-8de8-5638d480d32c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.411445 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.411374    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8054ec99-6079-4530-8de8-5638d480d32c-telemeter-client-tls\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.411664 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.411645    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8054ec99-6079-4530-8de8-5638d480d32c-federate-client-tls\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.427040 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.427017    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52qd6\" (UniqueName: \"kubernetes.io/projected/8054ec99-6079-4530-8de8-5638d480d32c-kube-api-access-52qd6\") pod \"telemeter-client-6f5dd465bf-nbfvs\" (UID: \"8054ec99-6079-4530-8de8-5638d480d32c\") " pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.517991 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.517960    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"
Jul 09 18:05:27.725773 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:27.725740    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs"]
Jul 09 18:05:27.732378 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:27.732341    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8054ec99_6079_4530_8de8_5638d480d32c.slice/crio-62d091a9953686190415f68d9bd88338dfb59527ad4b2b2fde94637728a8815b WatchSource:0}: Error finding container 62d091a9953686190415f68d9bd88338dfb59527ad4b2b2fde94637728a8815b: Status 404 returned error can't find the container with id 62d091a9953686190415f68d9bd88338dfb59527ad4b2b2fde94637728a8815b
Jul 09 18:05:28.139791 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.139584    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:05:28.142954 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.142928    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.152271 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.151533    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-debm0ta0g5r4b\""
Jul 09 18:05:28.152271 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.151817    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 09 18:05:28.152271 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.152123    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 09 18:05:28.152271 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.152166    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 09 18:05:28.152679 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.152303    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 18:05:28.153606 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.153046    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 09 18:05:28.153606 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.153239    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 09 18:05:28.153606 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.153327    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 09 18:05:28.153606 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.153416    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6x2k5\""
Jul 09 18:05:28.153606 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.153535    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 09 18:05:28.155818 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.155105    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 09 18:05:28.155818 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.155158    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 09 18:05:28.155818 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.155109    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 09 18:05:28.157636 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.157614    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 09 18:05:28.159940 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.159921    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 09 18:05:28.162888 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.162821    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 09 18:05:28.172068 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.172041    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:05:28.175824 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.175627    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 09 18:05:28.218942 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.218920    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-config\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219059 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.218947    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219059 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.218965    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjhjq\" (UniqueName: \"kubernetes.io/projected/10519a02-dcee-4479-93ab-0eb0332e4403-kube-api-access-tjhjq\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219059 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.218986    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219059 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219226 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219112    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10519a02-dcee-4479-93ab-0eb0332e4403-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219226 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219174    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219226 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219343 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219233    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219343 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219260    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219343 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219313    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219448 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219385    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219481 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219441    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-web-config\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219526 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219526 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219499    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219526 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219692 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219542    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10519a02-dcee-4479-93ab-0eb0332e4403-config-out\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219692 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219584    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219692 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.219692 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.219633    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.238521 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.238484    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-598db9cd7b-sfsh9" event={"ID":"094645b5-358b-4428-8564-296efebbbf21","Type":"ContainerStarted","Data":"af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253"}
Jul 09 18:05:28.239868 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.239846    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df" event={"ID":"d5d9c301-4af3-4b86-8c40-86876f728240","Type":"ContainerStarted","Data":"d19da163c5fdc0f578146d1a2ed77034980af5f0539f7c6cfea3e38270f8b60a"}
Jul 09 18:05:28.241078 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.241056    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs" event={"ID":"8054ec99-6079-4530-8de8-5638d480d32c","Type":"ContainerStarted","Data":"62d091a9953686190415f68d9bd88338dfb59527ad4b2b2fde94637728a8815b"}
Jul 09 18:05:28.291392 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.291342    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-598db9cd7b-sfsh9" podStartSLOduration=2.54261568 podStartE2EDuration="7.291323379s" totalImagesPullingTime="4.748707699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:28.290822251 +0000 UTC m=+65.129165235" watchObservedRunningTime="2026-07-09 18:05:28.291323379 +0000 UTC m=+65.129666367"
Jul 09 18:05:28.320699 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.320667    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.320699 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.320696    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.320873 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.320724    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.320873 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.320749    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.320873 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.320786    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.320873 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.320810    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.320873 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.320852    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-web-config\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.321489 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321449    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.321489 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321461    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.321489 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321487    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.321737 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321512    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.321737 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321576    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10519a02-dcee-4479-93ab-0eb0332e4403-config-out\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.321737 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321605    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.321737 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321655    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.321737 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321681    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.321737 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321724    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-config\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.322031 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321748    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.322031 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321777    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjhjq\" (UniqueName: \"kubernetes.io/projected/10519a02-dcee-4479-93ab-0eb0332e4403-kube-api-access-tjhjq\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.322031 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321824    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.322031 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321855    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.322031 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.321888    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10519a02-dcee-4479-93ab-0eb0332e4403-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.322278 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.322085    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.323042 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.322650    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.324171 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.323152    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.324171 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.323204    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.327003 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.326677    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.327003 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.326841    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.328154 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.327948    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-web-config\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.328529 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.328358    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.328745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.328664    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.328745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.328688    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.334631 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.330316    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10519a02-dcee-4479-93ab-0eb0332e4403-config-out\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.334631 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.330362    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.334631 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.330850    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.334631 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.331512    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.334631 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.333139    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10519a02-dcee-4479-93ab-0eb0332e4403-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.334631 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.333587    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-config\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.334631 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.333599    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.335399 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.335190    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.341065 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.341035    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjhjq\" (UniqueName: \"kubernetes.io/projected/10519a02-dcee-4479-93ab-0eb0332e4403-kube-api-access-tjhjq\") pod \"prometheus-k8s-0\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.459354 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.459110    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.523820 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.523013    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:05:28.523820 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.523064    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:05:28.527108 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.526721    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:28.527108 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.526961    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:28.539964 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.538504    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8695fc30-cdfd-4a63-bf30-33cf14224955-metrics-certs\") pod \"network-metrics-daemon-pw25w\" (UID: \"8695fc30-cdfd-4a63-bf30-33cf14224955\") " pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:05:28.541444 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.541407    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f42946-27c2-47be-b09b-cefcd2f8ce8e-original-pull-secret\") pod \"global-pull-secret-syncer-j6pks\" (UID: \"f0f42946-27c2-47be-b09b-cefcd2f8ce8e\") " pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:05:28.624978 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.624574    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5jk\" (UniqueName: \"kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk\") pod \"network-check-target-c9jnd\" (UID: \"d50047de-b0e9-4a3c-99ad-b4566e13172c\") " pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:05:28.628333 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.628138    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:28.638776 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.638754    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:28.639313 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.639290    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:05:28.649653 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.649628    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf5jk\" (UniqueName: \"kubernetes.io/projected/d50047de-b0e9-4a3c-99ad-b4566e13172c-kube-api-access-cf5jk\") pod \"network-check-target-c9jnd\" (UID: \"d50047de-b0e9-4a3c-99ad-b4566e13172c\") " pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:05:28.750218 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.750062    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mw97p\""
Jul 09 18:05:28.755943 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.755919    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6pks"
Jul 09 18:05:28.758063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.758040    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pw25w"
Jul 09 18:05:28.766637 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.766616    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-skcpw\""
Jul 09 18:05:28.773976 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:28.773955    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:05:28.932405 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:28.932371    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10519a02_dcee_4479_93ab_0eb0332e4403.slice/crio-4eae44eca99bde2252d169f089d2e7f1e8ea5b907db4f3e5ff846a8934c76638 WatchSource:0}: Error finding container 4eae44eca99bde2252d169f089d2e7f1e8ea5b907db4f3e5ff846a8934c76638: Status 404 returned error can't find the container with id 4eae44eca99bde2252d169f089d2e7f1e8ea5b907db4f3e5ff846a8934c76638
Jul 09 18:05:29.143375 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:29.143347    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c9jnd"]
Jul 09 18:05:29.146175 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:29.146143    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50047de_b0e9_4a3c_99ad_b4566e13172c.slice/crio-62c0bb6b347c8cb21bbcfc9648f8e08d39b954f95d7fa06c9592995496104ec8 WatchSource:0}: Error finding container 62c0bb6b347c8cb21bbcfc9648f8e08d39b954f95d7fa06c9592995496104ec8: Status 404 returned error can't find the container with id 62c0bb6b347c8cb21bbcfc9648f8e08d39b954f95d7fa06c9592995496104ec8
Jul 09 18:05:29.179090 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:29.179068    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2s9k4"
Jul 09 18:05:29.250135 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:29.250102    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c9jnd" event={"ID":"d50047de-b0e9-4a3c-99ad-b4566e13172c","Type":"ContainerStarted","Data":"62c0bb6b347c8cb21bbcfc9648f8e08d39b954f95d7fa06c9592995496104ec8"}
Jul 09 18:05:29.252007 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:29.251974    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df" event={"ID":"d5d9c301-4af3-4b86-8c40-86876f728240","Type":"ContainerStarted","Data":"f29702647f43ae558364f0eff006092a9724849bde0e5af0fa5fa0b9a116041f"}
Jul 09 18:05:29.252196 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:29.252143    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"
Jul 09 18:05:29.254053 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:29.254032    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerStarted","Data":"4eae44eca99bde2252d169f089d2e7f1e8ea5b907db4f3e5ff846a8934c76638"}
Jul 09 18:05:29.258041 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:29.257979    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df"
Jul 09 18:05:29.274793 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:29.274753    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s77df" podStartSLOduration=1.555707546 podStartE2EDuration="3.274738748s" totalImagesPullingTime="1.719031202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:29.273187477 +0000 UTC m=+66.111530462" watchObservedRunningTime="2026-07-09 18:05:29.274738748 +0000 UTC m=+66.113081736"
Jul 09 18:05:29.376915 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:29.376882    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pw25w"]
Jul 09 18:05:29.384650 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:29.384586    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j6pks"]
Jul 09 18:05:31.595446 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:31.595411    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:31.595908 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:31.595514    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:31.600905 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:31.600881    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:32.274934 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:32.274906    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:05:36.515460 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:36.515421    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8695fc30_cdfd_4a63_bf30_33cf14224955.slice/crio-d6be2448d8f49eb04e7cf7a4f81aeea5128c29bbab3127de06eb7513399db48a WatchSource:0}: Error finding container d6be2448d8f49eb04e7cf7a4f81aeea5128c29bbab3127de06eb7513399db48a: Status 404 returned error can't find the container with id d6be2448d8f49eb04e7cf7a4f81aeea5128c29bbab3127de06eb7513399db48a
Jul 09 18:05:36.516542 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:05:36.516183    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f42946_27c2_47be_b09b_cefcd2f8ce8e.slice/crio-f832b423a6651835b3a1166ea01944bfaa50dd6b7703272e9e4a52014a05d935 WatchSource:0}: Error finding container f832b423a6651835b3a1166ea01944bfaa50dd6b7703272e9e4a52014a05d935: Status 404 returned error can't find the container with id f832b423a6651835b3a1166ea01944bfaa50dd6b7703272e9e4a52014a05d935
Jul 09 18:05:37.290048 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:37.290011    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pw25w" event={"ID":"8695fc30-cdfd-4a63-bf30-33cf14224955","Type":"ContainerStarted","Data":"d6be2448d8f49eb04e7cf7a4f81aeea5128c29bbab3127de06eb7513399db48a"}
Jul 09 18:05:37.291487 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:37.291459    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j6pks" event={"ID":"f0f42946-27c2-47be-b09b-cefcd2f8ce8e","Type":"ContainerStarted","Data":"f832b423a6651835b3a1166ea01944bfaa50dd6b7703272e9e4a52014a05d935"}
Jul 09 18:05:38.301542 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:38.301458    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-df2bx" event={"ID":"da80f5f7-dfed-474e-96e7-c60902e0f08a","Type":"ContainerStarted","Data":"28ef452bc53bb1e57b9c24c88bfc6b5c107d98c5b12b492f9a77f12b8895d184"}
Jul 09 18:05:38.302046 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:38.302014    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-df2bx"
Jul 09 18:05:38.305447 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:38.305419    2572 generic.go:356] "Generic (PLEG): container finished" podID="10519a02-dcee-4479-93ab-0eb0332e4403" containerID="54076d536ede408f59398b3dfb69dbe73510ca9c6e4cf65696b2afad7ec2b4a8" exitCode=0
Jul 09 18:05:38.305599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:38.305511    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerDied","Data":"54076d536ede408f59398b3dfb69dbe73510ca9c6e4cf65696b2afad7ec2b4a8"}
Jul 09 18:05:38.309194 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:38.309129    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs" event={"ID":"8054ec99-6079-4530-8de8-5638d480d32c","Type":"ContainerStarted","Data":"47159ae019b1954b3470ee57021d5c821dd2bffccb45ff60ba6f1c477fc08cb9"}
Jul 09 18:05:38.309194 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:38.309179    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs" event={"ID":"8054ec99-6079-4530-8de8-5638d480d32c","Type":"ContainerStarted","Data":"1284bba6e6c2834682996a2cb44501c66636401701a97cbc4d1d42a57e456f03"}
Jul 09 18:05:38.310471 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:38.310434    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-df2bx"
Jul 09 18:05:38.323072 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:38.322928    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-df2bx" podStartSLOduration=2.491991934 podStartE2EDuration="23.32291095s" totalImagesPullingTime="20.830919016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:38.321880465 +0000 UTC m=+75.160223447" watchObservedRunningTime="2026-07-09 18:05:38.32291095 +0000 UTC m=+75.161253939"
Jul 09 18:05:41.332968 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:41.332858    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs" event={"ID":"8054ec99-6079-4530-8de8-5638d480d32c","Type":"ContainerStarted","Data":"4acce3f9825f8777a470367edf603abf34cd919b8062cdeb64f2f40a19db9691"}
Jul 09 18:05:41.335370 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:41.335117    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c9jnd" event={"ID":"d50047de-b0e9-4a3c-99ad-b4566e13172c","Type":"ContainerStarted","Data":"26adbd181507e3fc863c5490958c8d53c6d991bec69dfdd00b1ae24f8a206190"}
Jul 09 18:05:41.335920 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:41.335599    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:05:41.337796 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:41.337730    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pw25w" event={"ID":"8695fc30-cdfd-4a63-bf30-33cf14224955","Type":"ContainerStarted","Data":"9904ed3e566c698b9cd913527e6a29d3ea4f9a120ad45716ad121d70ea4299ab"}
Jul 09 18:05:41.360823 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:41.360546    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f5dd465bf-nbfvs" podStartSLOduration=4.136992512 podStartE2EDuration="14.36052835s" totalImagesPullingTime="10.223535838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:41.358854492 +0000 UTC m=+78.197197477" watchObservedRunningTime="2026-07-09 18:05:41.36052835 +0000 UTC m=+78.198871336"
Jul 09 18:05:41.380442 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:41.380095    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c9jnd" podStartSLOduration=65.657859205 podStartE2EDuration="1m17.38007648s" totalImagesPullingTime="11.722217275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:41.378926835 +0000 UTC m=+78.217269820" watchObservedRunningTime="2026-07-09 18:05:41.38007648 +0000 UTC m=+78.218419464"
Jul 09 18:05:45.365748 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:45.365713    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerStarted","Data":"6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968"}
Jul 09 18:05:45.368549 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:45.367836    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j6pks" event={"ID":"f0f42946-27c2-47be-b09b-cefcd2f8ce8e","Type":"ContainerStarted","Data":"1e52d1f28c5275a16cd97fae56e52fdd3970e0379da30df9f9fe75b6d5a35dba"}
Jul 09 18:05:45.371066 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:45.370955    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pw25w" event={"ID":"8695fc30-cdfd-4a63-bf30-33cf14224955","Type":"ContainerStarted","Data":"96d3b00b4015fb67d04b30320fb3e8da6aeb56bdcec56af35e76d118bfdbfccc"}
Jul 09 18:05:45.390369 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:45.390309    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j6pks" podStartSLOduration=72.757381963 podStartE2EDuration="1m21.390290166s" totalImagesPullingTime="8.632908203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:45.388995438 +0000 UTC m=+82.227338422" watchObservedRunningTime="2026-07-09 18:05:45.390290166 +0000 UTC m=+82.228633153"
Jul 09 18:05:45.409666 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:45.409608    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pw25w" podStartSLOduration=77.06897501 podStartE2EDuration="1m21.409592066s" totalImagesPullingTime="4.340617056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:45.409080471 +0000 UTC m=+82.247423466" watchObservedRunningTime="2026-07-09 18:05:45.409592066 +0000 UTC m=+82.247935049"
Jul 09 18:05:46.377926 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:46.377882    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerStarted","Data":"2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42"}
Jul 09 18:05:49.392324 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:49.392294    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerStarted","Data":"6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b"}
Jul 09 18:05:49.392611 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:49.392379    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerStarted","Data":"3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31"}
Jul 09 18:05:50.399698 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:50.399662    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerStarted","Data":"09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd"}
Jul 09 18:05:50.400068 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:50.399740    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerStarted","Data":"7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e"}
Jul 09 18:05:50.437177 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:50.437131    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9053354799999997 podStartE2EDuration="22.437116004s" totalImagesPullingTime="19.513821117s" totalInitContainerRuntime="17.959407ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:50.434380203 +0000 UTC m=+87.272723186" watchObservedRunningTime="2026-07-09 18:05:50.437116004 +0000 UTC m=+87.275459003"
Jul 09 18:05:52.886666 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:52.886632    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-598db9cd7b-sfsh9"]
Jul 09 18:05:53.459301 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:05:53.459269    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:13.355661 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:13.355630    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c9jnd"
Jul 09 18:06:17.907935 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:17.907811    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-598db9cd7b-sfsh9" podUID="094645b5-358b-4428-8564-296efebbbf21" containerName="console" containerID="cri-o://af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253" gracePeriod=15
Jul 09 18:06:18.185654 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.185546    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:06:18.234154 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.234114    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48lks\" (UniqueName: \"kubernetes.io/projected/094645b5-358b-4428-8564-296efebbbf21-kube-api-access-48lks\") pod \"094645b5-358b-4428-8564-296efebbbf21\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") "
Jul 09 18:06:18.234363 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.234169    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/094645b5-358b-4428-8564-296efebbbf21-console-oauth-config\") pod \"094645b5-358b-4428-8564-296efebbbf21\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") "
Jul 09 18:06:18.234363 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.234187    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/094645b5-358b-4428-8564-296efebbbf21-console-serving-cert\") pod \"094645b5-358b-4428-8564-296efebbbf21\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") "
Jul 09 18:06:18.234363 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.234206    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-oauth-serving-cert\") pod \"094645b5-358b-4428-8564-296efebbbf21\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") "
Jul 09 18:06:18.234363 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.234239    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-console-config\") pod \"094645b5-358b-4428-8564-296efebbbf21\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") "
Jul 09 18:06:18.234363 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.234266    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-service-ca\") pod \"094645b5-358b-4428-8564-296efebbbf21\" (UID: \"094645b5-358b-4428-8564-296efebbbf21\") "
Jul 09 18:06:18.234723 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.234692    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-console-config" (OuterVolumeSpecName: "console-config") pod "094645b5-358b-4428-8564-296efebbbf21" (UID: "094645b5-358b-4428-8564-296efebbbf21"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:18.234769 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.234721    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-service-ca" (OuterVolumeSpecName: "service-ca") pod "094645b5-358b-4428-8564-296efebbbf21" (UID: "094645b5-358b-4428-8564-296efebbbf21"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:18.234769 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.234700    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "094645b5-358b-4428-8564-296efebbbf21" (UID: "094645b5-358b-4428-8564-296efebbbf21"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:18.236647 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.236614    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094645b5-358b-4428-8564-296efebbbf21-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "094645b5-358b-4428-8564-296efebbbf21" (UID: "094645b5-358b-4428-8564-296efebbbf21"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:18.237157 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.237130    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094645b5-358b-4428-8564-296efebbbf21-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "094645b5-358b-4428-8564-296efebbbf21" (UID: "094645b5-358b-4428-8564-296efebbbf21"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:18.237209 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.237149    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094645b5-358b-4428-8564-296efebbbf21-kube-api-access-48lks" (OuterVolumeSpecName: "kube-api-access-48lks") pod "094645b5-358b-4428-8564-296efebbbf21" (UID: "094645b5-358b-4428-8564-296efebbbf21"). InnerVolumeSpecName "kube-api-access-48lks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:18.335385 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.335339    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-console-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:18.335385 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.335377    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-service-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:18.335385 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.335387    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48lks\" (UniqueName: \"kubernetes.io/projected/094645b5-358b-4428-8564-296efebbbf21-kube-api-access-48lks\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:18.335385 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.335398    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/094645b5-358b-4428-8564-296efebbbf21-console-oauth-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:18.335694 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.335407    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/094645b5-358b-4428-8564-296efebbbf21-console-serving-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:18.335694 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.335416    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/094645b5-358b-4428-8564-296efebbbf21-oauth-serving-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:18.508085 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.508050    2572 generic.go:356] "Generic (PLEG): container finished" podID="094645b5-358b-4428-8564-296efebbbf21" containerID="af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253" exitCode=0
Jul 09 18:06:18.508255 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.508106    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-598db9cd7b-sfsh9"
Jul 09 18:06:18.508255 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.508144    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-598db9cd7b-sfsh9" event={"ID":"094645b5-358b-4428-8564-296efebbbf21","Type":"ContainerDied","Data":"af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253"}
Jul 09 18:06:18.508255 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.508202    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-598db9cd7b-sfsh9" event={"ID":"094645b5-358b-4428-8564-296efebbbf21","Type":"ContainerDied","Data":"b488aeb2d3c9a89a728f9bed2e93dd7ea3e8e424d4f095757718bfb03d8f35ca"}
Jul 09 18:06:18.508255 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.508237    2572 scope.go:117] "RemoveContainer" containerID="af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253"
Jul 09 18:06:18.520030 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.520008    2572 scope.go:117] "RemoveContainer" containerID="af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253"
Jul 09 18:06:18.520381 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:06:18.520356    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253\": container with ID starting with af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253 not found: ID does not exist" containerID="af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253"
Jul 09 18:06:18.520434 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.520390    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253"} err="failed to get container status \"af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253\": rpc error: code = NotFound desc = could not find container \"af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253\": container with ID starting with af249768059dae86faca3a8d4151e0591662c3c0f6e9d91015841dfe1b211253 not found: ID does not exist"
Jul 09 18:06:18.529577 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.529528    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-598db9cd7b-sfsh9"]
Jul 09 18:06:18.535461 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:18.535433    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-598db9cd7b-sfsh9"]
Jul 09 18:06:19.932259 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:19.932227    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="094645b5-358b-4428-8564-296efebbbf21" path="/var/lib/kubelet/pods/094645b5-358b-4428-8564-296efebbbf21/volumes"
Jul 09 18:06:28.460152 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:28.460120    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:28.522038 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:28.522007    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:28.564853 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:28.564830    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.270220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.270184    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:46.270834 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.270499    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="prometheus" containerID="cri-o://6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968" gracePeriod=600
Jul 09 18:06:46.270834 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.270517    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="kube-rbac-proxy" containerID="cri-o://7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e" gracePeriod=600
Jul 09 18:06:46.270834 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.270603    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="kube-rbac-proxy-thanos" containerID="cri-o://09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd" gracePeriod=600
Jul 09 18:06:46.270834 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.270626    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="thanos-sidecar" containerID="cri-o://3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31" gracePeriod=600
Jul 09 18:06:46.270834 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.270654    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="config-reloader" containerID="cri-o://2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42" gracePeriod=600
Jul 09 18:06:46.270834 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.270603    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="kube-rbac-proxy-web" containerID="cri-o://6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b" gracePeriod=600
Jul 09 18:06:46.621002 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.620917    2572 generic.go:356] "Generic (PLEG): container finished" podID="10519a02-dcee-4479-93ab-0eb0332e4403" containerID="09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd" exitCode=0
Jul 09 18:06:46.621002 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.620944    2572 generic.go:356] "Generic (PLEG): container finished" podID="10519a02-dcee-4479-93ab-0eb0332e4403" containerID="7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e" exitCode=0
Jul 09 18:06:46.621002 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.620950    2572 generic.go:356] "Generic (PLEG): container finished" podID="10519a02-dcee-4479-93ab-0eb0332e4403" containerID="3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31" exitCode=0
Jul 09 18:06:46.621002 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.620956    2572 generic.go:356] "Generic (PLEG): container finished" podID="10519a02-dcee-4479-93ab-0eb0332e4403" containerID="2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42" exitCode=0
Jul 09 18:06:46.621002 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.620961    2572 generic.go:356] "Generic (PLEG): container finished" podID="10519a02-dcee-4479-93ab-0eb0332e4403" containerID="6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968" exitCode=0
Jul 09 18:06:46.621002 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.620988    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerDied","Data":"09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd"}
Jul 09 18:06:46.621285 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.621053    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerDied","Data":"7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e"}
Jul 09 18:06:46.621285 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.621080    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerDied","Data":"3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31"}
Jul 09 18:06:46.621285 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.621104    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerDied","Data":"2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42"}
Jul 09 18:06:46.621285 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:46.621126    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerDied","Data":"6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968"}
Jul 09 18:06:47.626305 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.626274    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:47.627354 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.627329    2572 generic.go:356] "Generic (PLEG): container finished" podID="10519a02-dcee-4479-93ab-0eb0332e4403" containerID="6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b" exitCode=0
Jul 09 18:06:47.627454 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.627380    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerDied","Data":"6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b"}
Jul 09 18:06:47.627454 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.627438    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10519a02-dcee-4479-93ab-0eb0332e4403","Type":"ContainerDied","Data":"4eae44eca99bde2252d169f089d2e7f1e8ea5b907db4f3e5ff846a8934c76638"}
Jul 09 18:06:47.627519 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.627477    2572 scope.go:117] "RemoveContainer" containerID="09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd"
Jul 09 18:06:47.635085 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.635066    2572 scope.go:117] "RemoveContainer" containerID="7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e"
Jul 09 18:06:47.642404 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.642384    2572 scope.go:117] "RemoveContainer" containerID="6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b"
Jul 09 18:06:47.649871 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.649852    2572 scope.go:117] "RemoveContainer" containerID="3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31"
Jul 09 18:06:47.657642 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.657618    2572 scope.go:117] "RemoveContainer" containerID="2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42"
Jul 09 18:06:47.666611 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.666581    2572 scope.go:117] "RemoveContainer" containerID="6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968"
Jul 09 18:06:47.677662 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.677636    2572 scope.go:117] "RemoveContainer" containerID="54076d536ede408f59398b3dfb69dbe73510ca9c6e4cf65696b2afad7ec2b4a8"
Jul 09 18:06:47.685607 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.685583    2572 scope.go:117] "RemoveContainer" containerID="09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd"
Jul 09 18:06:47.685946 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:06:47.685924    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd\": container with ID starting with 09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd not found: ID does not exist" containerID="09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd"
Jul 09 18:06:47.686021 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.685955    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd"} err="failed to get container status \"09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd\": rpc error: code = NotFound desc = could not find container \"09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd\": container with ID starting with 09b60b21b3cd7f6aa351812bf98888933b637055864f2a6ad8352e8d55992bcd not found: ID does not exist"
Jul 09 18:06:47.686021 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.685976    2572 scope.go:117] "RemoveContainer" containerID="7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e"
Jul 09 18:06:47.686275 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:06:47.686239    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e\": container with ID starting with 7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e not found: ID does not exist" containerID="7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e"
Jul 09 18:06:47.686313 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.686284    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e"} err="failed to get container status \"7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e\": rpc error: code = NotFound desc = could not find container \"7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e\": container with ID starting with 7acbaf8b0a7945d192348c42a0df7a6935dbb035cef21910be60786fc7508b6e not found: ID does not exist"
Jul 09 18:06:47.686313 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.686307    2572 scope.go:117] "RemoveContainer" containerID="6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b"
Jul 09 18:06:47.686589 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:06:47.686544    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b\": container with ID starting with 6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b not found: ID does not exist" containerID="6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b"
Jul 09 18:06:47.686692 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.686592    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b"} err="failed to get container status \"6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b\": rpc error: code = NotFound desc = could not find container \"6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b\": container with ID starting with 6691e6195ff1476a0d9a48b722731732481724d2790d424519d66e943844fd3b not found: ID does not exist"
Jul 09 18:06:47.686692 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.686608    2572 scope.go:117] "RemoveContainer" containerID="3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31"
Jul 09 18:06:47.686871 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:06:47.686855    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31\": container with ID starting with 3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31 not found: ID does not exist" containerID="3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31"
Jul 09 18:06:47.686918 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.686875    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31"} err="failed to get container status \"3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31\": rpc error: code = NotFound desc = could not find container \"3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31\": container with ID starting with 3f2a94956db767dc24c0a9400955d0cc7160ef4cbe3638b80c178d585fce9c31 not found: ID does not exist"
Jul 09 18:06:47.686918 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.686889    2572 scope.go:117] "RemoveContainer" containerID="2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42"
Jul 09 18:06:47.687149 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:06:47.687131    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42\": container with ID starting with 2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42 not found: ID does not exist" containerID="2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42"
Jul 09 18:06:47.687229 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.687153    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42"} err="failed to get container status \"2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42\": rpc error: code = NotFound desc = could not find container \"2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42\": container with ID starting with 2f14d08e2b19d7b942b529e2fcc691238b08c68ff8743b2d36d85bc63f856d42 not found: ID does not exist"
Jul 09 18:06:47.687229 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.687169    2572 scope.go:117] "RemoveContainer" containerID="6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968"
Jul 09 18:06:47.687412 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:06:47.687395    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968\": container with ID starting with 6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968 not found: ID does not exist" containerID="6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968"
Jul 09 18:06:47.687454 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.687417    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968"} err="failed to get container status \"6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968\": rpc error: code = NotFound desc = could not find container \"6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968\": container with ID starting with 6eeaa48bf88e18bc0cfd41e2a8c926db3c6192f9a221943c5c6fdb60760ba968 not found: ID does not exist"
Jul 09 18:06:47.687454 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.687433    2572 scope.go:117] "RemoveContainer" containerID="54076d536ede408f59398b3dfb69dbe73510ca9c6e4cf65696b2afad7ec2b4a8"
Jul 09 18:06:47.687712 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:06:47.687691    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54076d536ede408f59398b3dfb69dbe73510ca9c6e4cf65696b2afad7ec2b4a8\": container with ID starting with 54076d536ede408f59398b3dfb69dbe73510ca9c6e4cf65696b2afad7ec2b4a8 not found: ID does not exist" containerID="54076d536ede408f59398b3dfb69dbe73510ca9c6e4cf65696b2afad7ec2b4a8"
Jul 09 18:06:47.687806 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.687715    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54076d536ede408f59398b3dfb69dbe73510ca9c6e4cf65696b2afad7ec2b4a8"} err="failed to get container status \"54076d536ede408f59398b3dfb69dbe73510ca9c6e4cf65696b2afad7ec2b4a8\": rpc error: code = NotFound desc = could not find container \"54076d536ede408f59398b3dfb69dbe73510ca9c6e4cf65696b2afad7ec2b4a8\": container with ID starting with 54076d536ede408f59398b3dfb69dbe73510ca9c6e4cf65696b2afad7ec2b4a8 not found: ID does not exist"
Jul 09 18:06:47.766414 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766374    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-2\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.766621 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766423    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjhjq\" (UniqueName: \"kubernetes.io/projected/10519a02-dcee-4479-93ab-0eb0332e4403-kube-api-access-tjhjq\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.766621 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766486    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-db\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.766621 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766518    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-config\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.766621 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766541    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-thanos-prometheus-http-client-file\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.766621 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766601    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-trusted-ca-bundle\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.766817 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766635    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-kube-rbac-proxy\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.766817 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766668    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-web-config\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.766817 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766700    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-metrics-client-ca\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.766817 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766731    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-serving-certs-ca-bundle\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.766817 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766779    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.767013 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766818    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-grpc-tls\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.767013 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766844    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10519a02-dcee-4479-93ab-0eb0332e4403-config-out\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.767013 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766842    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:47.767013 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766871    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-metrics-client-certs\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.767013 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766898    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-1\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.767013 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766931    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-0\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.767013 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.766969    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10519a02-dcee-4479-93ab-0eb0332e4403-tls-assets\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.767013 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.767003    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-kubelet-serving-ca-bundle\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.767392 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.767030    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.767392 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.767062    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-tls\") pod \"10519a02-dcee-4479-93ab-0eb0332e4403\" (UID: \"10519a02-dcee-4479-93ab-0eb0332e4403\") "
Jul 09 18:06:47.767392 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.767277    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.767543 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.767393    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:47.767543 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.767429    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:47.769130 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.768125    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:06:47.769130 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.768193    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:47.769130 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.768890    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:47.769130 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.769051    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:47.769624 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.769572    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:47.769977 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.769931    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-config" (OuterVolumeSpecName: "config") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:47.770378 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.770351    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:47.770469 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.770393    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10519a02-dcee-4479-93ab-0eb0332e4403-kube-api-access-tjhjq" (OuterVolumeSpecName: "kube-api-access-tjhjq") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "kube-api-access-tjhjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:47.770613 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.770581    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:47.770824 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.770803    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:47.771022 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.770933    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:47.771604 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.771576    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10519a02-dcee-4479-93ab-0eb0332e4403-config-out" (OuterVolumeSpecName: "config-out") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:06:47.771882 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.771856    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10519a02-dcee-4479-93ab-0eb0332e4403-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:47.772538 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.772514    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:47.772672 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.772636    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:47.772920 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.772898    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:47.783008 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.782977    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-web-config" (OuterVolumeSpecName: "web-config") pod "10519a02-dcee-4479-93ab-0eb0332e4403" (UID: "10519a02-dcee-4479-93ab-0eb0332e4403"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:47.868710 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868652    2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868710 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868700    2572 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-grpc-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868710 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868716    2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10519a02-dcee-4479-93ab-0eb0332e4403-config-out\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868728    2572 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-metrics-client-certs\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868741    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868753    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868765    2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10519a02-dcee-4479-93ab-0eb0332e4403-tls-assets\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868776    2572 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868789    2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868801    2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868815    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjhjq\" (UniqueName: \"kubernetes.io/projected/10519a02-dcee-4479-93ab-0eb0332e4403-kube-api-access-tjhjq\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868826    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-k8s-db\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868839    2572 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868851    2572 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868864    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868877    2572 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-secret-kube-rbac-proxy\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868889    2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10519a02-dcee-4479-93ab-0eb0332e4403-web-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868900    2572 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-metrics-client-ca\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:47.868988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:47.868912    2572 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10519a02-dcee-4479-93ab-0eb0332e4403-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:06:48.631540 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.631501    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.654729 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.654631    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:48.661490 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.661448    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:48.688869 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.688830    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:48.689130 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689119    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="thanos-sidecar"
Jul 09 18:06:48.689173 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689134    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="thanos-sidecar"
Jul 09 18:06:48.689173 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689143    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="prometheus"
Jul 09 18:06:48.689173 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689149    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="prometheus"
Jul 09 18:06:48.689173 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689161    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="config-reloader"
Jul 09 18:06:48.689173 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689167    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="config-reloader"
Jul 09 18:06:48.689173 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689173    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="kube-rbac-proxy"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689179    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="kube-rbac-proxy"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689188    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094645b5-358b-4428-8564-296efebbbf21" containerName="console"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689193    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="094645b5-358b-4428-8564-296efebbbf21" containerName="console"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689205    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="init-config-reloader"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689211    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="init-config-reloader"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689218    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="kube-rbac-proxy-thanos"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689223    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="kube-rbac-proxy-thanos"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689230    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="kube-rbac-proxy-web"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689235    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="kube-rbac-proxy-web"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689276    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="thanos-sidecar"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689282    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="kube-rbac-proxy-thanos"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689288    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="config-reloader"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689294    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="prometheus"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689301    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="094645b5-358b-4428-8564-296efebbbf21" containerName="console"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689306    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="kube-rbac-proxy-web"
Jul 09 18:06:48.689339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.689311    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" containerName="kube-rbac-proxy"
Jul 09 18:06:48.693176 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.693149    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.696215 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.696186    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 09 18:06:48.696375 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.696240    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 09 18:06:48.696565 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.696537    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 09 18:06:48.696657 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.696573    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 09 18:06:48.696657 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.696601    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 09 18:06:48.696878 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.696851    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 09 18:06:48.696995 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.696978    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-debm0ta0g5r4b\""
Jul 09 18:06:48.697094 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.697075    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 09 18:06:48.697154 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.697079    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 09 18:06:48.697154 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.697080    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6x2k5\""
Jul 09 18:06:48.697252 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.697237    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 09 18:06:48.697303 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.697252    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 18:06:48.697471 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.697455    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 09 18:06:48.697471 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.697465    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 09 18:06:48.697646 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.697513    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 09 18:06:48.700944 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.700903    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 09 18:06:48.703788 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.703770    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 09 18:06:48.712474 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.712447    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:48.775269 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775227    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775269 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775267    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775477 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775293    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxrbb\" (UniqueName: \"kubernetes.io/projected/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-kube-api-access-gxrbb\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775477 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775346    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775477 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775477 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775384    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-config\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775477 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775416    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775477 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775431    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775477 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775452    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775477 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775467    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775740 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775483    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775740 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775502    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775740 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775547    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775740 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775618    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775740 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775642    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775740 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775679    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775740 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775696    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775740 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775723    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.775740 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775739    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.776045 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.775765    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.876813 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.876767    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.876813 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.876816    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.877064 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.876854    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.877064 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.876876    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.877064 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.876903    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.877064 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.876924    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.877064 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.876949    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.877064 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.876975    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.877064 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877000    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877598    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877656    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxrbb\" (UniqueName: \"kubernetes.io/projected/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-kube-api-access-gxrbb\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877690    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877701    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877743    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877773    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-config\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877786    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877805    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877831    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877859    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877886    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877913    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877944    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.878220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.877973    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.880061 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.878369    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.880061 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.879073    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.880061 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.879126    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.880647 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.880311    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.880647 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.880365    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.880647 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.880607    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.880856 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.880779    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.880909 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.880893    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-config\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.881449 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.881424    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.882261 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.882232    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.882400 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.882380    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.882692 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.882677    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.882791 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.882768    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.882874 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.882860    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.883240 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.883218    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.883427 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.883403    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:48.886132 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:48.886110    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxrbb\" (UniqueName: \"kubernetes.io/projected/5109a0a7-ca0e-4d03-9d14-7facb16c15fb-kube-api-access-gxrbb\") pod \"prometheus-k8s-0\" (UID: \"5109a0a7-ca0e-4d03-9d14-7facb16c15fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:49.006167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:49.006126    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:49.162075 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:49.162040    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:49.165662 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:06:49.165634    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5109a0a7_ca0e_4d03_9d14_7facb16c15fb.slice/crio-bc449d168a8f4b3412f8cc1bcb4a93b81e77796d2f1fa4cb393b8d5b430a6fb8 WatchSource:0}: Error finding container bc449d168a8f4b3412f8cc1bcb4a93b81e77796d2f1fa4cb393b8d5b430a6fb8: Status 404 returned error can't find the container with id bc449d168a8f4b3412f8cc1bcb4a93b81e77796d2f1fa4cb393b8d5b430a6fb8
Jul 09 18:06:49.636653 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:49.636620    2572 generic.go:356] "Generic (PLEG): container finished" podID="5109a0a7-ca0e-4d03-9d14-7facb16c15fb" containerID="ba7aab91e30a25355a3e72ef39469d550232348a2bb392002a85a18cd0045658" exitCode=0
Jul 09 18:06:49.637052 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:49.636714    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5109a0a7-ca0e-4d03-9d14-7facb16c15fb","Type":"ContainerDied","Data":"ba7aab91e30a25355a3e72ef39469d550232348a2bb392002a85a18cd0045658"}
Jul 09 18:06:49.637052 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:49.636776    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5109a0a7-ca0e-4d03-9d14-7facb16c15fb","Type":"ContainerStarted","Data":"bc449d168a8f4b3412f8cc1bcb4a93b81e77796d2f1fa4cb393b8d5b430a6fb8"}
Jul 09 18:06:49.932893 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:49.932864    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10519a02-dcee-4479-93ab-0eb0332e4403" path="/var/lib/kubelet/pods/10519a02-dcee-4479-93ab-0eb0332e4403/volumes"
Jul 09 18:06:50.644502 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:50.644464    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5109a0a7-ca0e-4d03-9d14-7facb16c15fb","Type":"ContainerStarted","Data":"481d2e48e987cee7d586f46d6cb7b073e36333d0e41ace836324e5c3c0f90243"}
Jul 09 18:06:50.644896 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:50.644538    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5109a0a7-ca0e-4d03-9d14-7facb16c15fb","Type":"ContainerStarted","Data":"2af412efbc61aeb1ab86edd4f4f6567dfca5ced0f511fb8fdda47efc11af67f3"}
Jul 09 18:06:50.644896 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:50.644594    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5109a0a7-ca0e-4d03-9d14-7facb16c15fb","Type":"ContainerStarted","Data":"fc7edece0a7c1d7288f2f6510a3b434e73b836f04d9e76a77f55a349dda83a87"}
Jul 09 18:06:50.644896 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:50.644622    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5109a0a7-ca0e-4d03-9d14-7facb16c15fb","Type":"ContainerStarted","Data":"e89cf31830d35430ee886eb73bdf7d4f7a47888798a6a11cf21d3cecc9f1482c"}
Jul 09 18:06:50.644896 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:50.644646    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5109a0a7-ca0e-4d03-9d14-7facb16c15fb","Type":"ContainerStarted","Data":"3640e30c5625895a97c5f5c4d9592f6ef7a98c676a6ecbbab007d26a24f31668"}
Jul 09 18:06:50.644896 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:50.644673    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5109a0a7-ca0e-4d03-9d14-7facb16c15fb","Type":"ContainerStarted","Data":"59aa4c8b2ca26d86b2c6b1ef731db5ebfc027ad0d7d879beea286648e917a3b5"}
Jul 09 18:06:50.677339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:50.677285    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.656769047 podStartE2EDuration="2.677264965s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.495918ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:50.674801004 +0000 UTC m=+147.513143989" watchObservedRunningTime="2026-07-09 18:06:50.677264965 +0000 UTC m=+147.515607949"
Jul 09 18:06:54.007079 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:06:54.007029    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:07:49.006543 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:07:49.006502    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:07:49.021861 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:07:49.021833    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:07:49.874154 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:07:49.874127    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:09:23.651019 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:09:23.650984    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:09:23.651540 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:09:23.651154    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:09:23.654932 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:09:23.654914    2572 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:10:22.356314 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.356285    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-tbb8g"]
Jul 09 18:10:22.359393 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.359372    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tbb8g"
Jul 09 18:10:22.362073 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.362055    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 18:10:22.362155 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.362135    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 18:10:22.363146 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.363128    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 18:10:22.363246 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.363150    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-fs24g\""
Jul 09 18:10:22.370103 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.370082    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tbb8g"]
Jul 09 18:10:22.538773 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.538733    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdflg\" (UniqueName: \"kubernetes.io/projected/7d48b26e-2420-4f74-ac50-407692b14644-kube-api-access-kdflg\") pod \"seaweedfs-64568bcd49-tbb8g\" (UID: \"7d48b26e-2420-4f74-ac50-407692b14644\") " pod="kserve/seaweedfs-64568bcd49-tbb8g"
Jul 09 18:10:22.538773 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.538780    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7d48b26e-2420-4f74-ac50-407692b14644-data\") pod \"seaweedfs-64568bcd49-tbb8g\" (UID: \"7d48b26e-2420-4f74-ac50-407692b14644\") " pod="kserve/seaweedfs-64568bcd49-tbb8g"
Jul 09 18:10:22.640215 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.640124    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7d48b26e-2420-4f74-ac50-407692b14644-data\") pod \"seaweedfs-64568bcd49-tbb8g\" (UID: \"7d48b26e-2420-4f74-ac50-407692b14644\") " pod="kserve/seaweedfs-64568bcd49-tbb8g"
Jul 09 18:10:22.640215 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.640210    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdflg\" (UniqueName: \"kubernetes.io/projected/7d48b26e-2420-4f74-ac50-407692b14644-kube-api-access-kdflg\") pod \"seaweedfs-64568bcd49-tbb8g\" (UID: \"7d48b26e-2420-4f74-ac50-407692b14644\") " pod="kserve/seaweedfs-64568bcd49-tbb8g"
Jul 09 18:10:22.640619 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.640598    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7d48b26e-2420-4f74-ac50-407692b14644-data\") pod \"seaweedfs-64568bcd49-tbb8g\" (UID: \"7d48b26e-2420-4f74-ac50-407692b14644\") " pod="kserve/seaweedfs-64568bcd49-tbb8g"
Jul 09 18:10:22.648809 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.648790    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdflg\" (UniqueName: \"kubernetes.io/projected/7d48b26e-2420-4f74-ac50-407692b14644-kube-api-access-kdflg\") pod \"seaweedfs-64568bcd49-tbb8g\" (UID: \"7d48b26e-2420-4f74-ac50-407692b14644\") " pod="kserve/seaweedfs-64568bcd49-tbb8g"
Jul 09 18:10:22.668787 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.668753    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tbb8g"
Jul 09 18:10:22.797293 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.797148    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tbb8g"]
Jul 09 18:10:22.799863 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:10:22.799832    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d48b26e_2420_4f74_ac50_407692b14644.slice/crio-4eb931435a749a8f07ae0f828ae5c39872b995c1fbe17a86960375c13ff7a0bb WatchSource:0}: Error finding container 4eb931435a749a8f07ae0f828ae5c39872b995c1fbe17a86960375c13ff7a0bb: Status 404 returned error can't find the container with id 4eb931435a749a8f07ae0f828ae5c39872b995c1fbe17a86960375c13ff7a0bb
Jul 09 18:10:22.801152 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:22.801137    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:10:23.412860 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:23.412824    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tbb8g" event={"ID":"7d48b26e-2420-4f74-ac50-407692b14644","Type":"ContainerStarted","Data":"4eb931435a749a8f07ae0f828ae5c39872b995c1fbe17a86960375c13ff7a0bb"}
Jul 09 18:10:25.430756 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:25.430732    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 18:10:26.425543 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:26.425503    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tbb8g" event={"ID":"7d48b26e-2420-4f74-ac50-407692b14644","Type":"ContainerStarted","Data":"3653646eccc1bef16a0c03f628b1cd963470b0141ce443c930ac124add8e6d80"}
Jul 09 18:10:26.425739 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:26.425612    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-tbb8g"
Jul 09 18:10:26.442931 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:26.442882    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-tbb8g" podStartSLOduration=1.817697877 podStartE2EDuration="4.44286819s" totalImagesPullingTime="2.625170313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:26.441097725 +0000 UTC m=+363.279440712" watchObservedRunningTime="2026-07-09 18:10:26.44286819 +0000 UTC m=+363.281211173"
Jul 09 18:10:32.430740 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:32.430708    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-tbb8g"
Jul 09 18:10:55.599904 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.599855    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-847895f65b-qzdwj"]
Jul 09 18:10:55.603890 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.603861    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.607335 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.607312    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6txwz\""
Jul 09 18:10:55.607497 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.607370    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 18:10:55.607497 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.607390    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 18:10:55.607694 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.607670    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 18:10:55.607782 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.607760    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 18:10:55.607842 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.607823    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 18:10:55.615023 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.615004    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 18:10:55.618071 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.618050    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-847895f65b-qzdwj"]
Jul 09 18:10:55.679594 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.679569    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070eb4f2-3b5e-4c4b-ade8-5a818c225987-trusted-ca-bundle\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.679692 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.679610    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/070eb4f2-3b5e-4c4b-ade8-5a818c225987-console-serving-cert\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.679692 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.679632    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/070eb4f2-3b5e-4c4b-ade8-5a818c225987-oauth-serving-cert\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.679692 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.679647    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slcdj\" (UniqueName: \"kubernetes.io/projected/070eb4f2-3b5e-4c4b-ade8-5a818c225987-kube-api-access-slcdj\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.679692 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.679678    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/070eb4f2-3b5e-4c4b-ade8-5a818c225987-service-ca\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.679825 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.679751    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/070eb4f2-3b5e-4c4b-ade8-5a818c225987-console-config\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.679825 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.679787    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/070eb4f2-3b5e-4c4b-ade8-5a818c225987-console-oauth-config\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.780456 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.780429    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/070eb4f2-3b5e-4c4b-ade8-5a818c225987-console-config\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.780544 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.780464    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/070eb4f2-3b5e-4c4b-ade8-5a818c225987-console-oauth-config\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.780544 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.780494    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070eb4f2-3b5e-4c4b-ade8-5a818c225987-trusted-ca-bundle\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.780544 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.780530    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/070eb4f2-3b5e-4c4b-ade8-5a818c225987-console-serving-cert\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.780673 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.780581    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/070eb4f2-3b5e-4c4b-ade8-5a818c225987-oauth-serving-cert\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.780713 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.780701    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slcdj\" (UniqueName: \"kubernetes.io/projected/070eb4f2-3b5e-4c4b-ade8-5a818c225987-kube-api-access-slcdj\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.780771 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.780754    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/070eb4f2-3b5e-4c4b-ade8-5a818c225987-service-ca\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.781298 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.781276    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/070eb4f2-3b5e-4c4b-ade8-5a818c225987-console-config\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.781432 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.781412    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070eb4f2-3b5e-4c4b-ade8-5a818c225987-trusted-ca-bundle\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.781497 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.781364    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/070eb4f2-3b5e-4c4b-ade8-5a818c225987-oauth-serving-cert\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.781497 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.781376    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/070eb4f2-3b5e-4c4b-ade8-5a818c225987-service-ca\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.783158 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.783136    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/070eb4f2-3b5e-4c4b-ade8-5a818c225987-console-serving-cert\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.783267 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.783199    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/070eb4f2-3b5e-4c4b-ade8-5a818c225987-console-oauth-config\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.789330 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.789307    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slcdj\" (UniqueName: \"kubernetes.io/projected/070eb4f2-3b5e-4c4b-ade8-5a818c225987-kube-api-access-slcdj\") pod \"console-847895f65b-qzdwj\" (UID: \"070eb4f2-3b5e-4c4b-ade8-5a818c225987\") " pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:55.915069 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:55.915010    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:10:56.042907 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:56.042864    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-847895f65b-qzdwj"]
Jul 09 18:10:56.045591 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:10:56.045565    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070eb4f2_3b5e_4c4b_ade8_5a818c225987.slice/crio-eb36046c9d1ea50026524142d2aeba6b341ddab9825ffea2b78cf63e6e17aa93 WatchSource:0}: Error finding container eb36046c9d1ea50026524142d2aeba6b341ddab9825ffea2b78cf63e6e17aa93: Status 404 returned error can't find the container with id eb36046c9d1ea50026524142d2aeba6b341ddab9825ffea2b78cf63e6e17aa93
Jul 09 18:10:56.541817 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:56.541775    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847895f65b-qzdwj" event={"ID":"070eb4f2-3b5e-4c4b-ade8-5a818c225987","Type":"ContainerStarted","Data":"60a8f23d0e7fc8c029fb4c2ed681f361960e8b11dcdba4e1e42f0bdc3c047d01"}
Jul 09 18:10:56.542005 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:56.541867    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847895f65b-qzdwj" event={"ID":"070eb4f2-3b5e-4c4b-ade8-5a818c225987","Type":"ContainerStarted","Data":"eb36046c9d1ea50026524142d2aeba6b341ddab9825ffea2b78cf63e6e17aa93"}
Jul 09 18:10:56.563181 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:10:56.563135    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-847895f65b-qzdwj" podStartSLOduration=1.563117127 podStartE2EDuration="1.563117127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:56.561892294 +0000 UTC m=+393.400235279" watchObservedRunningTime="2026-07-09 18:10:56.563117127 +0000 UTC m=+393.401460111"
Jul 09 18:11:05.915200 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:05.915155    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:11:05.915682 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:05.915237    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:11:05.920034 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:05.920010    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:11:06.583317 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:06.583276    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-847895f65b-qzdwj"
Jul 09 18:11:50.102617 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:50.102519    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-9pzs7"]
Jul 09 18:11:50.105304 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:50.105287    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9pzs7"
Jul 09 18:11:50.115097 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:50.115071    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9pzs7"]
Jul 09 18:11:50.219715 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:50.219683    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jsg8\" (UniqueName: \"kubernetes.io/projected/5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982-kube-api-access-6jsg8\") pod \"s3-init-9pzs7\" (UID: \"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982\") " pod="kserve/s3-init-9pzs7"
Jul 09 18:11:50.219882 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:50.219749    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982-model-cache\") pod \"s3-init-9pzs7\" (UID: \"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982\") " pod="kserve/s3-init-9pzs7"
Jul 09 18:11:50.321030 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:50.320995    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jsg8\" (UniqueName: \"kubernetes.io/projected/5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982-kube-api-access-6jsg8\") pod \"s3-init-9pzs7\" (UID: \"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982\") " pod="kserve/s3-init-9pzs7"
Jul 09 18:11:50.321198 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:50.321046    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982-model-cache\") pod \"s3-init-9pzs7\" (UID: \"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982\") " pod="kserve/s3-init-9pzs7"
Jul 09 18:11:50.321422 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:50.321395    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982-model-cache\") pod \"s3-init-9pzs7\" (UID: \"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982\") " pod="kserve/s3-init-9pzs7"
Jul 09 18:11:50.330861 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:50.330827    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jsg8\" (UniqueName: \"kubernetes.io/projected/5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982-kube-api-access-6jsg8\") pod \"s3-init-9pzs7\" (UID: \"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982\") " pod="kserve/s3-init-9pzs7"
Jul 09 18:11:50.424074 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:50.423991    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9pzs7"
Jul 09 18:11:50.564451 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:50.564423    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9pzs7"]
Jul 09 18:11:50.566756 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:11:50.566725    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ccc6e2a_d2e9_4a3b_90c1_a6b2aae3b982.slice/crio-9e198ccf8b7240cbcd50de49352284d44c7ff2255f12af32a0198ed8622b2c59 WatchSource:0}: Error finding container 9e198ccf8b7240cbcd50de49352284d44c7ff2255f12af32a0198ed8622b2c59: Status 404 returned error can't find the container with id 9e198ccf8b7240cbcd50de49352284d44c7ff2255f12af32a0198ed8622b2c59
Jul 09 18:11:50.746167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:50.746130    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9pzs7" event={"ID":"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982","Type":"ContainerStarted","Data":"9e198ccf8b7240cbcd50de49352284d44c7ff2255f12af32a0198ed8622b2c59"}
Jul 09 18:11:54.772701 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:54.772664    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9pzs7" event={"ID":"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982","Type":"ContainerStarted","Data":"4d601a140f75fd735025c967b588edf6e6b6f9f23588f1ce639ef91c855df3f6"}
Jul 09 18:11:56.782497 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:56.782463    2572 generic.go:356] "Generic (PLEG): container finished" podID="5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982" containerID="4d601a140f75fd735025c967b588edf6e6b6f9f23588f1ce639ef91c855df3f6" exitCode=0
Jul 09 18:11:56.782917 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:11:56.782539    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9pzs7" event={"ID":"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982","Type":"ContainerDied","Data":"4d601a140f75fd735025c967b588edf6e6b6f9f23588f1ce639ef91c855df3f6"}
Jul 09 18:12:01.808809 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:12:01.808779    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9pzs7" event={"ID":"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982","Type":"ContainerStarted","Data":"daf9ae655bd8ac1b913cf51bd4ea3528f2003d8708971c5f5f964cfb02a3a052"}
Jul 09 18:12:01.827978 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:12:01.827931    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-9pzs7" podStartSLOduration=1.700078639 podStartE2EDuration="11.827917676s" totalImagesPullingTime="8.434209827s" totalInitContainerRuntime="1.69362921s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:12:01.827467902 +0000 UTC m=+458.665810887" watchObservedRunningTime="2026-07-09 18:12:01.827917676 +0000 UTC m=+458.666260659"
Jul 09 18:13:07.050302 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:07.050270    2572 generic.go:356] "Generic (PLEG): container finished" podID="5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982" containerID="daf9ae655bd8ac1b913cf51bd4ea3528f2003d8708971c5f5f964cfb02a3a052" exitCode=0
Jul 09 18:13:07.050698 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:07.050359    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9pzs7" event={"ID":"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982","Type":"ContainerDied","Data":"daf9ae655bd8ac1b913cf51bd4ea3528f2003d8708971c5f5f964cfb02a3a052"}
Jul 09 18:13:08.180960 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:08.180939    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9pzs7"
Jul 09 18:13:08.262397 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:08.262363    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jsg8\" (UniqueName: \"kubernetes.io/projected/5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982-kube-api-access-6jsg8\") pod \"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982\" (UID: \"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982\") "
Jul 09 18:13:08.262591 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:08.262417    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982-model-cache\") pod \"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982\" (UID: \"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982\") "
Jul 09 18:13:08.264506 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:08.264481    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982-kube-api-access-6jsg8" (OuterVolumeSpecName: "kube-api-access-6jsg8") pod "5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982" (UID: "5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982"). InnerVolumeSpecName "kube-api-access-6jsg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:13:08.281089 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:08.281056    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982" (UID: "5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:13:08.363352 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:08.363289    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982-model-cache\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:13:08.363352 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:08.363311    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jsg8\" (UniqueName: \"kubernetes.io/projected/5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982-kube-api-access-6jsg8\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:13:09.059259 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:09.059232    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9pzs7" event={"ID":"5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982","Type":"ContainerDied","Data":"9e198ccf8b7240cbcd50de49352284d44c7ff2255f12af32a0198ed8622b2c59"}
Jul 09 18:13:09.059259 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:09.059266    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e198ccf8b7240cbcd50de49352284d44c7ff2255f12af32a0198ed8622b2c59"
Jul 09 18:13:09.059462 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:09.059268    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9pzs7"
Jul 09 18:13:21.223550 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.223518    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"]
Jul 09 18:13:21.223971 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.223834    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982" containerName="s3-init"
Jul 09 18:13:21.223971 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.223858    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982" containerName="s3-init"
Jul 09 18:13:21.223971 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.223870    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982" containerName="download-hf-model"
Jul 09 18:13:21.223971 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.223875    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982" containerName="download-hf-model"
Jul 09 18:13:21.223971 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.223926    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982" containerName="s3-init"
Jul 09 18:13:21.226852 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.226834    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.229514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.229490    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:13:21.229659 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.229540    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:13:21.229659 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.229648    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:13:21.229777 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.229664    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:13:21.230745 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.230727    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fgcf4\""
Jul 09 18:13:21.230823 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.230730    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 09 18:13:21.230823 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.230733    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 09 18:13:21.240730 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.240711    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"]
Jul 09 18:13:21.342104 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.342073    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"]
Jul 09 18:13:21.345330 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.345315    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:21.349379 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.349354    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7eb51-predictor-serving-cert\""
Jul 09 18:13:21.349469 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.349360    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7eb51-kube-rbac-proxy-sar-config\""
Jul 09 18:13:21.359755 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.359736    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"]
Jul 09 18:13:21.367681 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.367662    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b04426a-61f4-42e6-9c88-f6057862a1c5-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.367784 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.367695    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b04426a-61f4-42e6-9c88-f6057862a1c5-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.367784 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.367721    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b04426a-61f4-42e6-9c88-f6057862a1c5-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.367887 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.367813    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b04426a-61f4-42e6-9c88-f6057862a1c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.367887 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.367847    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdk2x\" (UniqueName: \"kubernetes.io/projected/4b04426a-61f4-42e6-9c88-f6057862a1c5-kube-api-access-wdk2x\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.469238 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.469213    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b04426a-61f4-42e6-9c88-f6057862a1c5-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.469374 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.469256    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b04426a-61f4-42e6-9c88-f6057862a1c5-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.469374 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.469286    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b04426a-61f4-42e6-9c88-f6057862a1c5-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.469374 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.469314    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b829581f-d2ea-4d3d-b726-d2763dac62eb-proxy-tls\") pod \"success-200-isvc-7eb51-predictor-57447d7988-rk8g9\" (UID: \"b829581f-d2ea-4d3d-b726-d2763dac62eb\") " pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:21.469374 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.469356    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b04426a-61f4-42e6-9c88-f6057862a1c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.469640 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.469614    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfntg\" (UniqueName: \"kubernetes.io/projected/b829581f-d2ea-4d3d-b726-d2763dac62eb-kube-api-access-nfntg\") pod \"success-200-isvc-7eb51-predictor-57447d7988-rk8g9\" (UID: \"b829581f-d2ea-4d3d-b726-d2763dac62eb\") " pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:21.469731 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.469633    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b04426a-61f4-42e6-9c88-f6057862a1c5-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.469731 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.469665    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdk2x\" (UniqueName: \"kubernetes.io/projected/4b04426a-61f4-42e6-9c88-f6057862a1c5-kube-api-access-wdk2x\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.469731 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.469702    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b829581f-d2ea-4d3d-b726-d2763dac62eb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7eb51-predictor-57447d7988-rk8g9\" (UID: \"b829581f-d2ea-4d3d-b726-d2763dac62eb\") " pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:21.470123 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.470098    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b04426a-61f4-42e6-9c88-f6057862a1c5-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.470282 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.470262    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b04426a-61f4-42e6-9c88-f6057862a1c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.471891 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.471875    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b04426a-61f4-42e6-9c88-f6057862a1c5-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.484006 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.483954    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdk2x\" (UniqueName: \"kubernetes.io/projected/4b04426a-61f4-42e6-9c88-f6057862a1c5-kube-api-access-wdk2x\") pod \"isvc-sklearn-graph-1-predictor-b95b6754-55kpx\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.537514 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.537491    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:21.564337 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.564313    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"]
Jul 09 18:13:21.567831 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.567814    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.570193 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.570169    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfntg\" (UniqueName: \"kubernetes.io/projected/b829581f-d2ea-4d3d-b726-d2763dac62eb-kube-api-access-nfntg\") pod \"success-200-isvc-7eb51-predictor-57447d7988-rk8g9\" (UID: \"b829581f-d2ea-4d3d-b726-d2763dac62eb\") " pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:21.570305 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.570218    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b829581f-d2ea-4d3d-b726-d2763dac62eb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7eb51-predictor-57447d7988-rk8g9\" (UID: \"b829581f-d2ea-4d3d-b726-d2763dac62eb\") " pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:21.570372 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.570325    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b829581f-d2ea-4d3d-b726-d2763dac62eb-proxy-tls\") pod \"success-200-isvc-7eb51-predictor-57447d7988-rk8g9\" (UID: \"b829581f-d2ea-4d3d-b726-d2763dac62eb\") " pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:21.570589 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.570541    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 09 18:13:21.570589 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.570576    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 09 18:13:21.571103 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.571085    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b829581f-d2ea-4d3d-b726-d2763dac62eb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7eb51-predictor-57447d7988-rk8g9\" (UID: \"b829581f-d2ea-4d3d-b726-d2763dac62eb\") " pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:21.572766 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.572740    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b829581f-d2ea-4d3d-b726-d2763dac62eb-proxy-tls\") pod \"success-200-isvc-7eb51-predictor-57447d7988-rk8g9\" (UID: \"b829581f-d2ea-4d3d-b726-d2763dac62eb\") " pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:21.580710 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.580686    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfntg\" (UniqueName: \"kubernetes.io/projected/b829581f-d2ea-4d3d-b726-d2763dac62eb-kube-api-access-nfntg\") pod \"success-200-isvc-7eb51-predictor-57447d7988-rk8g9\" (UID: \"b829581f-d2ea-4d3d-b726-d2763dac62eb\") " pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:21.592532 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.592422    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"]
Jul 09 18:13:21.654256 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.654231    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:21.671464 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.671441    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.671593 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.671482    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.671663 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.671617    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.671722 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.671680    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xv67\" (UniqueName: \"kubernetes.io/projected/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kube-api-access-2xv67\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.671778 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.671719    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.679959 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.679941    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"]
Jul 09 18:13:21.682687 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:13:21.682666    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b04426a_61f4_42e6_9c88_f6057862a1c5.slice/crio-abe6e99832d52d582635b43cb761348c3123b362f7e82a3f0179a6a68c7f2066 WatchSource:0}: Error finding container abe6e99832d52d582635b43cb761348c3123b362f7e82a3f0179a6a68c7f2066: Status 404 returned error can't find the container with id abe6e99832d52d582635b43cb761348c3123b362f7e82a3f0179a6a68c7f2066
Jul 09 18:13:21.772793 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.772772    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.772914 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.772811    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.772914 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.772857    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.772914 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.772893    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xv67\" (UniqueName: \"kubernetes.io/projected/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kube-api-access-2xv67\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.773084 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.772922    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.773800 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.773320    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.773800 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.773619    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.774369 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.774268    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.775723 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.775705    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.783197 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.783173    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xv67\" (UniqueName: \"kubernetes.io/projected/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kube-api-access-2xv67\") pod \"isvc-xgboost-graph-predictor-555fc69d56-mb2dk\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.802738 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.802718    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"]
Jul 09 18:13:21.807972 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:13:21.807951    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb829581f_d2ea_4d3d_b726_d2763dac62eb.slice/crio-514ff12dc8325e652638b052a08bc59eefcb7491956245a5530b9874d2abe216 WatchSource:0}: Error finding container 514ff12dc8325e652638b052a08bc59eefcb7491956245a5530b9874d2abe216: Status 404 returned error can't find the container with id 514ff12dc8325e652638b052a08bc59eefcb7491956245a5530b9874d2abe216
Jul 09 18:13:21.885151 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.885121    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:21.950550 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.950522    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"]
Jul 09 18:13:21.954515 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.954495    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:21.960217 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.960196    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 09 18:13:21.960343 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.960241    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 09 18:13:21.999833 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:21.999772    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"]
Jul 09 18:13:22.058580 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.058534    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"]
Jul 09 18:13:22.061786 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:13:22.061761    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dc2f6c_13e3_4bac_b4f8_6acb2755397c.slice/crio-419783842e0d7d8946c8e2a4d981b264d20bd4f735f94b204ba7c31edc4bd54f WatchSource:0}: Error finding container 419783842e0d7d8946c8e2a4d981b264d20bd4f735f94b204ba7c31edc4bd54f: Status 404 returned error can't find the container with id 419783842e0d7d8946c8e2a4d981b264d20bd4f735f94b204ba7c31edc4bd54f
Jul 09 18:13:22.075656 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.075626    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.075779 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.075757    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.075821 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.075795    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2drqb\" (UniqueName: \"kubernetes.io/projected/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kube-api-access-2drqb\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.075869 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.075841    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.075932 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.075910    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.107969 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.107948    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" event={"ID":"b829581f-d2ea-4d3d-b726-d2763dac62eb","Type":"ContainerStarted","Data":"514ff12dc8325e652638b052a08bc59eefcb7491956245a5530b9874d2abe216"}
Jul 09 18:13:22.109057 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.109034    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" event={"ID":"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c","Type":"ContainerStarted","Data":"419783842e0d7d8946c8e2a4d981b264d20bd4f735f94b204ba7c31edc4bd54f"}
Jul 09 18:13:22.110545 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.110522    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" event={"ID":"4b04426a-61f4-42e6-9c88-f6057862a1c5","Type":"ContainerStarted","Data":"9e723f31f697bc01ef0b94570df432be90b51dc33778e5628cafb976cb7f8386"}
Jul 09 18:13:22.110655 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.110597    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" event={"ID":"4b04426a-61f4-42e6-9c88-f6057862a1c5","Type":"ContainerStarted","Data":"abe6e99832d52d582635b43cb761348c3123b362f7e82a3f0179a6a68c7f2066"}
Jul 09 18:13:22.176997 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.176968    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.177121 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.177013    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2drqb\" (UniqueName: \"kubernetes.io/projected/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kube-api-access-2drqb\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.177121 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.177084    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.177234 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.177136    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.177329 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.177280    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.177797 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.177770    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.177933 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.177916    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.178043 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.178022    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.179749 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.179726    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.188353 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.188328    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2drqb\" (UniqueName: \"kubernetes.io/projected/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kube-api-access-2drqb\") pod \"isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.264762 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.264682    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:22.397962 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:22.397920    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"]
Jul 09 18:13:22.401351 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:13:22.401024    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88647b4_dbbf_477a_a61c_eb28a0df5ffb.slice/crio-18aae8c1bcd77f131ffc2070cd37a757107461c9c37a12b05cf2f94cc24c2870 WatchSource:0}: Error finding container 18aae8c1bcd77f131ffc2070cd37a757107461c9c37a12b05cf2f94cc24c2870: Status 404 returned error can't find the container with id 18aae8c1bcd77f131ffc2070cd37a757107461c9c37a12b05cf2f94cc24c2870
Jul 09 18:13:23.119970 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:23.119933    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" event={"ID":"f88647b4-dbbf-477a-a61c-eb28a0df5ffb","Type":"ContainerStarted","Data":"2e0c020a4723320f0c314ba061ae9f16dbf66f02eb22c9c470c928ddae5c19a7"}
Jul 09 18:13:23.120157 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:23.120009    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" event={"ID":"f88647b4-dbbf-477a-a61c-eb28a0df5ffb","Type":"ContainerStarted","Data":"18aae8c1bcd77f131ffc2070cd37a757107461c9c37a12b05cf2f94cc24c2870"}
Jul 09 18:13:23.125677 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:23.124644    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" event={"ID":"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c","Type":"ContainerStarted","Data":"5e9decdbf1d2931ad857ad8b31631666ab33594373d0638d2d2cdfb04e435d7c"}
Jul 09 18:13:27.145569 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:27.145529    2572 generic.go:356] "Generic (PLEG): container finished" podID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerID="2e0c020a4723320f0c314ba061ae9f16dbf66f02eb22c9c470c928ddae5c19a7" exitCode=0
Jul 09 18:13:27.145983 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:27.145619    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" event={"ID":"f88647b4-dbbf-477a-a61c-eb28a0df5ffb","Type":"ContainerDied","Data":"2e0c020a4723320f0c314ba061ae9f16dbf66f02eb22c9c470c928ddae5c19a7"}
Jul 09 18:13:27.147853 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:27.147810    2572 generic.go:356] "Generic (PLEG): container finished" podID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerID="5e9decdbf1d2931ad857ad8b31631666ab33594373d0638d2d2cdfb04e435d7c" exitCode=0
Jul 09 18:13:27.147930 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:27.147891    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" event={"ID":"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c","Type":"ContainerDied","Data":"5e9decdbf1d2931ad857ad8b31631666ab33594373d0638d2d2cdfb04e435d7c"}
Jul 09 18:13:27.155280 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:27.155254    2572 generic.go:356] "Generic (PLEG): container finished" podID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerID="9e723f31f697bc01ef0b94570df432be90b51dc33778e5628cafb976cb7f8386" exitCode=0
Jul 09 18:13:27.155463 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:27.155334    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" event={"ID":"4b04426a-61f4-42e6-9c88-f6057862a1c5","Type":"ContainerDied","Data":"9e723f31f697bc01ef0b94570df432be90b51dc33778e5628cafb976cb7f8386"}
Jul 09 18:13:53.317673 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:53.317646    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" event={"ID":"f88647b4-dbbf-477a-a61c-eb28a0df5ffb","Type":"ContainerStarted","Data":"65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16"}
Jul 09 18:13:53.319436 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:53.319411    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" event={"ID":"4b04426a-61f4-42e6-9c88-f6057862a1c5","Type":"ContainerStarted","Data":"a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373"}
Jul 09 18:13:53.320647 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:53.320609    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" event={"ID":"b829581f-d2ea-4d3d-b726-d2763dac62eb","Type":"ContainerStarted","Data":"525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b"}
Jul 09 18:13:54.346052 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:54.345967    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" event={"ID":"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c","Type":"ContainerStarted","Data":"bce7203d8879f3e58c538fdc73250b6218e333692b77bb5d38d5d5bb5c340f77"}
Jul 09 18:13:57.362009 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.361973    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" event={"ID":"b829581f-d2ea-4d3d-b726-d2763dac62eb","Type":"ContainerStarted","Data":"73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5"}
Jul 09 18:13:57.362444 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.362202    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:57.362444 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.362354    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:13:57.363665 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.363640    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:13:57.364252 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.364231    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" event={"ID":"f88647b4-dbbf-477a-a61c-eb28a0df5ffb","Type":"ContainerStarted","Data":"753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff"}
Jul 09 18:13:57.364327 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.364292    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:57.364389 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.364327    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:13:57.365477 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.365453    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:13:57.366367 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.366347    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" event={"ID":"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c","Type":"ContainerStarted","Data":"b86a08d2abfea72e02df52815750eb0fc14f5cddddc84c824a54af1bf2f00571"}
Jul 09 18:13:57.366585 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.366570    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:57.366670 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.366610    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:13:57.367709 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.367681    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:13:57.368483 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.368467    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" event={"ID":"4b04426a-61f4-42e6-9c88-f6057862a1c5","Type":"ContainerStarted","Data":"e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a"}
Jul 09 18:13:57.368612 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.368591    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:57.368677 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.368645    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:13:57.369462 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.369441    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:13:57.386731 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.386691    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" podStartSLOduration=1.716421248 podStartE2EDuration="36.386677878s" totalImagesPullingTime="34.67025663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:57.383950922 +0000 UTC m=+574.222293906" watchObservedRunningTime="2026-07-09 18:13:57.386677878 +0000 UTC m=+574.225020862"
Jul 09 18:13:57.407255 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.407221    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podStartSLOduration=3.518579432 podStartE2EDuration="36.407209397s" totalImagesPullingTime="29.269491804s" totalInitContainerRuntime="3.619138161s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:57.405950472 +0000 UTC m=+574.244293456" watchObservedRunningTime="2026-07-09 18:13:57.407209397 +0000 UTC m=+574.245552380"
Jul 09 18:13:57.434577 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.433789    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" podStartSLOduration=3.160503469 podStartE2EDuration="36.433769353s" totalImagesPullingTime="29.307266942s" totalInitContainerRuntime="3.965998942s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:57.432189815 +0000 UTC m=+574.270532825" watchObservedRunningTime="2026-07-09 18:13:57.433769353 +0000 UTC m=+574.272112339"
Jul 09 18:13:57.453328 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:57.453249    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podStartSLOduration=3.175400208 podStartE2EDuration="36.453234755s" totalImagesPullingTime="29.31528419s" totalInitContainerRuntime="3.962550357s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:57.452486571 +0000 UTC m=+574.290829555" watchObservedRunningTime="2026-07-09 18:13:57.453234755 +0000 UTC m=+574.291577740"
Jul 09 18:13:58.373214 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:58.373174    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:13:58.373699 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:58.373184    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:13:58.373699 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:58.373184    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:13:58.373813 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:13:58.373701    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:14:03.378214 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:03.378182    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:14:03.378744 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:03.378288    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:14:03.378967 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:03.378943    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:14:03.378967 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:03.378942    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:14:03.379094 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:03.378959    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:14:03.379094 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:03.379068    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:14:03.379424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:03.379401    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:14:03.379703 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:03.379685    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:14:13.379678 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:13.379637    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:14:13.380135 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:13.379643    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:14:13.380135 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:13.379646    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:14:13.380135 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:13.379649    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:14:23.379396 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:23.379358    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:14:23.379823 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:23.379369    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:14:23.379823 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:23.379372    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:14:23.379823 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:23.379616    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:14:23.677784 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:23.677706    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:14:23.678324 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:23.678302    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:14:33.379110 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:33.379074    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:14:33.379490 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:33.379076    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:14:33.379490 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:33.379315    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:14:33.380497 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:33.380476    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:14:43.379433 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:43.379392    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:14:43.379893 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:43.379400    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:14:43.379893 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:43.379696    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:14:43.379893 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:43.379707    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:14:53.379425 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:53.379348    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:14:53.380035 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:53.379700    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:14:53.380035 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:14:53.379703    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:15:01.141000 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.140972    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"]
Jul 09 18:15:01.144539 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.144519    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:01.147132 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.147106    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-7eb51-serving-cert\""
Jul 09 18:15:01.147237 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.147106    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-7eb51-kube-rbac-proxy-sar-config\""
Jul 09 18:15:01.153651 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.153632    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"]
Jul 09 18:15:01.285510 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.285478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d36a391-f930-438b-b9a6-6297e5d30764-openshift-service-ca-bundle\") pod \"switch-graph-7eb51-75dbc6577f-5fksf\" (UID: \"0d36a391-f930-438b-b9a6-6297e5d30764\") " pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:01.285636 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.285525    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d36a391-f930-438b-b9a6-6297e5d30764-proxy-tls\") pod \"switch-graph-7eb51-75dbc6577f-5fksf\" (UID: \"0d36a391-f930-438b-b9a6-6297e5d30764\") " pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:01.386318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.386292    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d36a391-f930-438b-b9a6-6297e5d30764-openshift-service-ca-bundle\") pod \"switch-graph-7eb51-75dbc6577f-5fksf\" (UID: \"0d36a391-f930-438b-b9a6-6297e5d30764\") " pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:01.386441 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.386335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d36a391-f930-438b-b9a6-6297e5d30764-proxy-tls\") pod \"switch-graph-7eb51-75dbc6577f-5fksf\" (UID: \"0d36a391-f930-438b-b9a6-6297e5d30764\") " pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:01.386891 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.386870    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d36a391-f930-438b-b9a6-6297e5d30764-openshift-service-ca-bundle\") pod \"switch-graph-7eb51-75dbc6577f-5fksf\" (UID: \"0d36a391-f930-438b-b9a6-6297e5d30764\") " pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:01.388654 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.388635    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d36a391-f930-438b-b9a6-6297e5d30764-proxy-tls\") pod \"switch-graph-7eb51-75dbc6577f-5fksf\" (UID: \"0d36a391-f930-438b-b9a6-6297e5d30764\") " pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:01.454704 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.454685    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:01.582678 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.582655    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"]
Jul 09 18:15:01.585106 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:15:01.585079    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d36a391_f930_438b_b9a6_6297e5d30764.slice/crio-9eafc1dbbdded367c613c6725564d08d9abba460c594b241404b85db0fbb728b WatchSource:0}: Error finding container 9eafc1dbbdded367c613c6725564d08d9abba460c594b241404b85db0fbb728b: Status 404 returned error can't find the container with id 9eafc1dbbdded367c613c6725564d08d9abba460c594b241404b85db0fbb728b
Jul 09 18:15:01.641596 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:01.641565    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf" event={"ID":"0d36a391-f930-438b-b9a6-6297e5d30764","Type":"ContainerStarted","Data":"9eafc1dbbdded367c613c6725564d08d9abba460c594b241404b85db0fbb728b"}
Jul 09 18:15:03.379696 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:03.379655    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:15:03.379696 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:03.379686    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:15:04.656095 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:04.656010    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf" event={"ID":"0d36a391-f930-438b-b9a6-6297e5d30764","Type":"ContainerStarted","Data":"152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404"}
Jul 09 18:15:04.656436 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:04.656105    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:04.678161 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:04.678118    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf" podStartSLOduration=1.052557159 podStartE2EDuration="3.6781048s" totalImagesPullingTime="2.625547641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:04.675684239 +0000 UTC m=+641.514027226" watchObservedRunningTime="2026-07-09 18:15:04.6781048 +0000 UTC m=+641.516447784"
Jul 09 18:15:05.937359 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:05.937328    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:15:10.666035 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:10.666006    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:11.345087 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.345057    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"]
Jul 09 18:15:11.345261 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.345224    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf" podUID="0d36a391-f930-438b-b9a6-6297e5d30764" containerName="switch-graph-7eb51" containerID="cri-o://152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404" gracePeriod=30
Jul 09 18:15:11.586911 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.586879    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"]
Jul 09 18:15:11.587185 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.587148    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kserve-container" containerID="cri-o://525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b" gracePeriod=30
Jul 09 18:15:11.587185 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.587171    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kube-rbac-proxy" containerID="cri-o://73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5" gracePeriod=30
Jul 09 18:15:11.642564 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.642525    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"]
Jul 09 18:15:11.653988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.653964    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:11.656385 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.656360    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ea8a9-kube-rbac-proxy-sar-config\""
Jul 09 18:15:11.656536 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.656516    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ea8a9-predictor-serving-cert\""
Jul 09 18:15:11.657748 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.657707    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"]
Jul 09 18:15:11.776857 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.776831    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98b183e2-8018-400f-adca-4853efd8cd70-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") " pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:11.777182 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.776862    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8789p\" (UniqueName: \"kubernetes.io/projected/98b183e2-8018-400f-adca-4853efd8cd70-kube-api-access-8789p\") pod \"success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") " pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:11.777182 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.776891    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98b183e2-8018-400f-adca-4853efd8cd70-proxy-tls\") pod \"success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") " pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:11.877787 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.877712    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98b183e2-8018-400f-adca-4853efd8cd70-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") " pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:11.877787 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.877746    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8789p\" (UniqueName: \"kubernetes.io/projected/98b183e2-8018-400f-adca-4853efd8cd70-kube-api-access-8789p\") pod \"success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") " pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:11.877787 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.877777    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98b183e2-8018-400f-adca-4853efd8cd70-proxy-tls\") pod \"success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") " pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:11.878030 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:11.877890    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-serving-cert: secret "success-200-isvc-ea8a9-predictor-serving-cert" not found
Jul 09 18:15:11.878030 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:11.877947    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98b183e2-8018-400f-adca-4853efd8cd70-proxy-tls podName:98b183e2-8018-400f-adca-4853efd8cd70 nodeName:}" failed. No retries permitted until 2026-07-09 18:15:12.377929799 +0000 UTC m=+649.216272778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/98b183e2-8018-400f-adca-4853efd8cd70-proxy-tls") pod "success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" (UID: "98b183e2-8018-400f-adca-4853efd8cd70") : secret "success-200-isvc-ea8a9-predictor-serving-cert" not found
Jul 09 18:15:11.878399 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.878379    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98b183e2-8018-400f-adca-4853efd8cd70-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") " pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:11.888806 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:11.888783    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8789p\" (UniqueName: \"kubernetes.io/projected/98b183e2-8018-400f-adca-4853efd8cd70-kube-api-access-8789p\") pod \"success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") " pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:12.382046 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:12.382017    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98b183e2-8018-400f-adca-4853efd8cd70-proxy-tls\") pod \"success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") " pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:12.384227 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:12.384209    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98b183e2-8018-400f-adca-4853efd8cd70-proxy-tls\") pod \"success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") " pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:12.565820 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:12.565787    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:12.692329 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:12.692300    2572 generic.go:356] "Generic (PLEG): container finished" podID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerID="73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5" exitCode=2
Jul 09 18:15:12.692440 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:12.692353    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" event={"ID":"b829581f-d2ea-4d3d-b726-d2763dac62eb","Type":"ContainerDied","Data":"73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5"}
Jul 09 18:15:12.698295 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:12.698274    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"]
Jul 09 18:15:12.700518 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:15:12.700493    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b183e2_8018_400f_adca_4853efd8cd70.slice/crio-990e552990c186f9f0d1996e60e34f18d01e30c48e74c07e0b6c3e678248d22e WatchSource:0}: Error finding container 990e552990c186f9f0d1996e60e34f18d01e30c48e74c07e0b6c3e678248d22e: Status 404 returned error can't find the container with id 990e552990c186f9f0d1996e60e34f18d01e30c48e74c07e0b6c3e678248d22e
Jul 09 18:15:13.373889 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:13.373851    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 09 18:15:13.379258 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:13.379230    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:15:13.380193 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:13.380174    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:15:13.698146 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:13.698111    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" event={"ID":"98b183e2-8018-400f-adca-4853efd8cd70","Type":"ContainerStarted","Data":"9571edb4d15bac7b869e322c39e88d1fae3c4702fe52045943fef6e7ae74d207"}
Jul 09 18:15:13.698308 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:13.698178    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" event={"ID":"98b183e2-8018-400f-adca-4853efd8cd70","Type":"ContainerStarted","Data":"90c9561b2fa832a0f84871c147d329b48a9b82756a1082e6647b29e5fe0d192a"}
Jul 09 18:15:13.698308 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:13.698207    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" event={"ID":"98b183e2-8018-400f-adca-4853efd8cd70","Type":"ContainerStarted","Data":"990e552990c186f9f0d1996e60e34f18d01e30c48e74c07e0b6c3e678248d22e"}
Jul 09 18:15:13.698308 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:13.698238    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:13.721467 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:13.721411    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" podStartSLOduration=2.721395438 podStartE2EDuration="2.721395438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:13.720526541 +0000 UTC m=+650.558869528" watchObservedRunningTime="2026-07-09 18:15:13.721395438 +0000 UTC m=+650.559738423"
Jul 09 18:15:14.433988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.433965    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:15:14.599734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.599675    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b829581f-d2ea-4d3d-b726-d2763dac62eb-kube-rbac-proxy-sar-config\") pod \"b829581f-d2ea-4d3d-b726-d2763dac62eb\" (UID: \"b829581f-d2ea-4d3d-b726-d2763dac62eb\") "
Jul 09 18:15:14.599868 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.599763    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b829581f-d2ea-4d3d-b726-d2763dac62eb-proxy-tls\") pod \"b829581f-d2ea-4d3d-b726-d2763dac62eb\" (UID: \"b829581f-d2ea-4d3d-b726-d2763dac62eb\") "
Jul 09 18:15:14.599938 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.599921    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfntg\" (UniqueName: \"kubernetes.io/projected/b829581f-d2ea-4d3d-b726-d2763dac62eb-kube-api-access-nfntg\") pod \"b829581f-d2ea-4d3d-b726-d2763dac62eb\" (UID: \"b829581f-d2ea-4d3d-b726-d2763dac62eb\") "
Jul 09 18:15:14.600090 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.600067    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b829581f-d2ea-4d3d-b726-d2763dac62eb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b829581f-d2ea-4d3d-b726-d2763dac62eb" (UID: "b829581f-d2ea-4d3d-b726-d2763dac62eb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:15:14.600223 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.600207    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b829581f-d2ea-4d3d-b726-d2763dac62eb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:14.601807 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.601774    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b829581f-d2ea-4d3d-b726-d2763dac62eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b829581f-d2ea-4d3d-b726-d2763dac62eb" (UID: "b829581f-d2ea-4d3d-b726-d2763dac62eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:15:14.601901 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.601848    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b829581f-d2ea-4d3d-b726-d2763dac62eb-kube-api-access-nfntg" (OuterVolumeSpecName: "kube-api-access-nfntg") pod "b829581f-d2ea-4d3d-b726-d2763dac62eb" (UID: "b829581f-d2ea-4d3d-b726-d2763dac62eb"). InnerVolumeSpecName "kube-api-access-nfntg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:15:14.701122 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.701098    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b829581f-d2ea-4d3d-b726-d2763dac62eb-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:14.701210 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.701125    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfntg\" (UniqueName: \"kubernetes.io/projected/b829581f-d2ea-4d3d-b726-d2763dac62eb-kube-api-access-nfntg\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:14.704601 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.704578    2572 generic.go:356] "Generic (PLEG): container finished" podID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerID="525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b" exitCode=0
Jul 09 18:15:14.704683 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.704622    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"
Jul 09 18:15:14.704683 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.704654    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" event={"ID":"b829581f-d2ea-4d3d-b726-d2763dac62eb","Type":"ContainerDied","Data":"525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b"}
Jul 09 18:15:14.704744 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.704721    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9" event={"ID":"b829581f-d2ea-4d3d-b726-d2763dac62eb","Type":"ContainerDied","Data":"514ff12dc8325e652638b052a08bc59eefcb7491956245a5530b9874d2abe216"}
Jul 09 18:15:14.704788 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.704758    2572 scope.go:117] "RemoveContainer" containerID="73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5"
Jul 09 18:15:14.705213 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.705138    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:14.706364 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.706337    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 18:15:14.713922 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.713907    2572 scope.go:117] "RemoveContainer" containerID="525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b"
Jul 09 18:15:14.721075 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.721057    2572 scope.go:117] "RemoveContainer" containerID="73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5"
Jul 09 18:15:14.721318 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:14.721301    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5\": container with ID starting with 73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5 not found: ID does not exist" containerID="73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5"
Jul 09 18:15:14.721364 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.721326    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5"} err="failed to get container status \"73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5\": rpc error: code = NotFound desc = could not find container \"73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5\": container with ID starting with 73626a36dae00989d66fee3a310e04f76c83a71f3bc51584a35f5ab295e239d5 not found: ID does not exist"
Jul 09 18:15:14.721364 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.721343    2572 scope.go:117] "RemoveContainer" containerID="525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b"
Jul 09 18:15:14.721590 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:14.721568    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b\": container with ID starting with 525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b not found: ID does not exist" containerID="525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b"
Jul 09 18:15:14.721637 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.721599    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b"} err="failed to get container status \"525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b\": rpc error: code = NotFound desc = could not find container \"525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b\": container with ID starting with 525229924c0367ad7dd16b8901bed6076991868c936939759d01925ae80ed60b not found: ID does not exist"
Jul 09 18:15:14.728114 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.728090    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"]
Jul 09 18:15:14.731616 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:14.731597    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7eb51-predictor-57447d7988-rk8g9"]
Jul 09 18:15:15.664724 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:15.664690    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf" podUID="0d36a391-f930-438b-b9a6-6297e5d30764" containerName="switch-graph-7eb51" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:15.710605 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:15.710569    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 18:15:15.934170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:15.934104    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" path="/var/lib/kubelet/pods/b829581f-d2ea-4d3d-b726-d2763dac62eb/volumes"
Jul 09 18:15:20.664032 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:20.663994    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf" podUID="0d36a391-f930-438b-b9a6-6297e5d30764" containerName="switch-graph-7eb51" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:20.714603 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:20.714582    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:15:20.715039 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:20.715003    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 18:15:25.664224 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:25.664184    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf" podUID="0d36a391-f930-438b-b9a6-6297e5d30764" containerName="switch-graph-7eb51" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:25.664681 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:25.664272    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:30.664334 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:30.664299    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf" podUID="0d36a391-f930-438b-b9a6-6297e5d30764" containerName="switch-graph-7eb51" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:30.715419 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:30.715384    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 18:15:31.092963 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.092928    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"]
Jul 09 18:15:31.093274 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.093262    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kube-rbac-proxy"
Jul 09 18:15:31.093325 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.093275    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kube-rbac-proxy"
Jul 09 18:15:31.093325 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.093283    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kserve-container"
Jul 09 18:15:31.093325 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.093289    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kserve-container"
Jul 09 18:15:31.093427 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.093352    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kserve-container"
Jul 09 18:15:31.093427 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.093362    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b829581f-d2ea-4d3d-b726-d2763dac62eb" containerName="kube-rbac-proxy"
Jul 09 18:15:31.097441 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.097422    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:15:31.100335 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.100316    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 09 18:15:31.100428 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.100321    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 09 18:15:31.107350 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.107324    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"]
Jul 09 18:15:31.117843 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.117825    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33474294-5f25-48ad-8fb9-13904d12dd23-openshift-service-ca-bundle\") pod \"model-chainer-5b4bb9966-sq87t\" (UID: \"33474294-5f25-48ad-8fb9-13904d12dd23\") " pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:15:31.117932 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.117852    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33474294-5f25-48ad-8fb9-13904d12dd23-proxy-tls\") pod \"model-chainer-5b4bb9966-sq87t\" (UID: \"33474294-5f25-48ad-8fb9-13904d12dd23\") " pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:15:31.218254 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.218233    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33474294-5f25-48ad-8fb9-13904d12dd23-openshift-service-ca-bundle\") pod \"model-chainer-5b4bb9966-sq87t\" (UID: \"33474294-5f25-48ad-8fb9-13904d12dd23\") " pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:15:31.218347 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.218264    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33474294-5f25-48ad-8fb9-13904d12dd23-proxy-tls\") pod \"model-chainer-5b4bb9966-sq87t\" (UID: \"33474294-5f25-48ad-8fb9-13904d12dd23\") " pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:15:31.218388 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:31.218374    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 09 18:15:31.218430 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:31.218421    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33474294-5f25-48ad-8fb9-13904d12dd23-proxy-tls podName:33474294-5f25-48ad-8fb9-13904d12dd23 nodeName:}" failed. No retries permitted until 2026-07-09 18:15:31.718407125 +0000 UTC m=+668.556750086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/33474294-5f25-48ad-8fb9-13904d12dd23-proxy-tls") pod "model-chainer-5b4bb9966-sq87t" (UID: "33474294-5f25-48ad-8fb9-13904d12dd23") : secret "model-chainer-serving-cert" not found
Jul 09 18:15:31.218837 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.218821    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33474294-5f25-48ad-8fb9-13904d12dd23-openshift-service-ca-bundle\") pod \"model-chainer-5b4bb9966-sq87t\" (UID: \"33474294-5f25-48ad-8fb9-13904d12dd23\") " pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:15:31.722660 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.722630    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33474294-5f25-48ad-8fb9-13904d12dd23-proxy-tls\") pod \"model-chainer-5b4bb9966-sq87t\" (UID: \"33474294-5f25-48ad-8fb9-13904d12dd23\") " pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:15:31.725027 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:31.724998    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33474294-5f25-48ad-8fb9-13904d12dd23-proxy-tls\") pod \"model-chainer-5b4bb9966-sq87t\" (UID: \"33474294-5f25-48ad-8fb9-13904d12dd23\") " pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:15:32.007979 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:32.007917    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:15:32.133164 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:32.133142    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"]
Jul 09 18:15:32.135210 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:15:32.135185    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33474294_5f25_48ad_8fb9_13904d12dd23.slice/crio-a75fb82d09dfd4dfcdc4d4843c8ea6ff610c21fc35a3a5da4bd9633159ee3ec7 WatchSource:0}: Error finding container a75fb82d09dfd4dfcdc4d4843c8ea6ff610c21fc35a3a5da4bd9633159ee3ec7: Status 404 returned error can't find the container with id a75fb82d09dfd4dfcdc4d4843c8ea6ff610c21fc35a3a5da4bd9633159ee3ec7
Jul 09 18:15:32.783176 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:32.783144    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t" event={"ID":"33474294-5f25-48ad-8fb9-13904d12dd23","Type":"ContainerStarted","Data":"ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3"}
Jul 09 18:15:32.783571 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:32.783192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t" event={"ID":"33474294-5f25-48ad-8fb9-13904d12dd23","Type":"ContainerStarted","Data":"a75fb82d09dfd4dfcdc4d4843c8ea6ff610c21fc35a3a5da4bd9633159ee3ec7"}
Jul 09 18:15:32.783571 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:32.783211    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:15:32.804781 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:32.804739    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t" podStartSLOduration=1.8047249029999999 podStartE2EDuration="1.804724903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:32.803527826 +0000 UTC m=+669.641870810" watchObservedRunningTime="2026-07-09 18:15:32.804724903 +0000 UTC m=+669.643067887"
Jul 09 18:15:35.664060 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:35.664025    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf" podUID="0d36a391-f930-438b-b9a6-6297e5d30764" containerName="switch-graph-7eb51" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:38.792431 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:38.792387    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:15:40.664249 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:40.664215    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf" podUID="0d36a391-f930-438b-b9a6-6297e5d30764" containerName="switch-graph-7eb51" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:40.715067 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:40.715034    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 18:15:41.183232 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.183202    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"]
Jul 09 18:15:41.183434 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.183359    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t" podUID="33474294-5f25-48ad-8fb9-13904d12dd23" containerName="model-chainer" containerID="cri-o://ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3" gracePeriod=30
Jul 09 18:15:41.344039 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.344009    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"]
Jul 09 18:15:41.344254 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.344229    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container" containerID="cri-o://a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373" gracePeriod=30
Jul 09 18:15:41.344330 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.344266    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kube-rbac-proxy" containerID="cri-o://e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a" gracePeriod=30
Jul 09 18:15:41.380860 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.380833    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"]
Jul 09 18:15:41.384421 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.384399    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:41.388033 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.388015    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-04bd4-predictor-serving-cert\""
Jul 09 18:15:41.388119 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.388052    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-04bd4-kube-rbac-proxy-sar-config\""
Jul 09 18:15:41.394699 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.394678    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"]
Jul 09 18:15:41.414833 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.414813    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"]
Jul 09 18:15:41.415033 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.415009    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kserve-container" containerID="cri-o://bce7203d8879f3e58c538fdc73250b6218e333692b77bb5d38d5d5bb5c340f77" gracePeriod=30
Jul 09 18:15:41.415098 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.415034    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kube-rbac-proxy" containerID="cri-o://b86a08d2abfea72e02df52815750eb0fc14f5cddddc84c824a54af1bf2f00571" gracePeriod=30
Jul 09 18:15:41.492963 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.492928    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhkd7\" (UniqueName: \"kubernetes.io/projected/ac367207-f257-4d07-9ad4-6f7f5ca22489-kube-api-access-rhkd7\") pod \"success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw\" (UID: \"ac367207-f257-4d07-9ad4-6f7f5ca22489\") " pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:41.493129 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.492981    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac367207-f257-4d07-9ad4-6f7f5ca22489-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw\" (UID: \"ac367207-f257-4d07-9ad4-6f7f5ca22489\") " pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:41.493129 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.493092    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac367207-f257-4d07-9ad4-6f7f5ca22489-proxy-tls\") pod \"success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw\" (UID: \"ac367207-f257-4d07-9ad4-6f7f5ca22489\") " pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:41.522595 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.522346    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"]
Jul 09 18:15:41.522711 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.522640    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container" containerID="cri-o://65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16" gracePeriod=30
Jul 09 18:15:41.522979 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.522953    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kube-rbac-proxy" containerID="cri-o://753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff" gracePeriod=30
Jul 09 18:15:41.593809 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.593782    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac367207-f257-4d07-9ad4-6f7f5ca22489-proxy-tls\") pod \"success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw\" (UID: \"ac367207-f257-4d07-9ad4-6f7f5ca22489\") " pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:41.593904 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.593863    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhkd7\" (UniqueName: \"kubernetes.io/projected/ac367207-f257-4d07-9ad4-6f7f5ca22489-kube-api-access-rhkd7\") pod \"success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw\" (UID: \"ac367207-f257-4d07-9ad4-6f7f5ca22489\") " pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:41.593904 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.593894    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac367207-f257-4d07-9ad4-6f7f5ca22489-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw\" (UID: \"ac367207-f257-4d07-9ad4-6f7f5ca22489\") " pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:41.594508 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.594489    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac367207-f257-4d07-9ad4-6f7f5ca22489-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw\" (UID: \"ac367207-f257-4d07-9ad4-6f7f5ca22489\") " pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:41.596028 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.596009    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac367207-f257-4d07-9ad4-6f7f5ca22489-proxy-tls\") pod \"success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw\" (UID: \"ac367207-f257-4d07-9ad4-6f7f5ca22489\") " pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:41.602016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.601995    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhkd7\" (UniqueName: \"kubernetes.io/projected/ac367207-f257-4d07-9ad4-6f7f5ca22489-kube-api-access-rhkd7\") pod \"success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw\" (UID: \"ac367207-f257-4d07-9ad4-6f7f5ca22489\") " pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:41.636922 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.636898    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:41.695107 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.695086    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d36a391-f930-438b-b9a6-6297e5d30764-openshift-service-ca-bundle\") pod \"0d36a391-f930-438b-b9a6-6297e5d30764\" (UID: \"0d36a391-f930-438b-b9a6-6297e5d30764\") "
Jul 09 18:15:41.695368 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.695126    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d36a391-f930-438b-b9a6-6297e5d30764-proxy-tls\") pod \"0d36a391-f930-438b-b9a6-6297e5d30764\" (UID: \"0d36a391-f930-438b-b9a6-6297e5d30764\") "
Jul 09 18:15:41.695426 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.695405    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d36a391-f930-438b-b9a6-6297e5d30764-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0d36a391-f930-438b-b9a6-6297e5d30764" (UID: "0d36a391-f930-438b-b9a6-6297e5d30764"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:15:41.696866 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.696841    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d36a391-f930-438b-b9a6-6297e5d30764-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d36a391-f930-438b-b9a6-6297e5d30764" (UID: "0d36a391-f930-438b-b9a6-6297e5d30764"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:15:41.731091 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.731028    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:41.796422 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.796397    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d36a391-f930-438b-b9a6-6297e5d30764-openshift-service-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:41.796422 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.796424    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d36a391-f930-438b-b9a6-6297e5d30764-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:41.820332 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.820297    2572 generic.go:356] "Generic (PLEG): container finished" podID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerID="753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff" exitCode=2
Jul 09 18:15:41.820439 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.820335    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" event={"ID":"f88647b4-dbbf-477a-a61c-eb28a0df5ffb","Type":"ContainerDied","Data":"753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff"}
Jul 09 18:15:41.823841 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.821661    2572 generic.go:356] "Generic (PLEG): container finished" podID="0d36a391-f930-438b-b9a6-6297e5d30764" containerID="152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404" exitCode=0
Jul 09 18:15:41.823841 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.821796    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"
Jul 09 18:15:41.823841 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.822794    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf" event={"ID":"0d36a391-f930-438b-b9a6-6297e5d30764","Type":"ContainerDied","Data":"152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404"}
Jul 09 18:15:41.823841 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.822845    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf" event={"ID":"0d36a391-f930-438b-b9a6-6297e5d30764","Type":"ContainerDied","Data":"9eafc1dbbdded367c613c6725564d08d9abba460c594b241404b85db0fbb728b"}
Jul 09 18:15:41.823841 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.822874    2572 scope.go:117] "RemoveContainer" containerID="152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404"
Jul 09 18:15:41.828000 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.827973    2572 generic.go:356] "Generic (PLEG): container finished" podID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerID="b86a08d2abfea72e02df52815750eb0fc14f5cddddc84c824a54af1bf2f00571" exitCode=2
Jul 09 18:15:41.828105 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.828053    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" event={"ID":"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c","Type":"ContainerDied","Data":"b86a08d2abfea72e02df52815750eb0fc14f5cddddc84c824a54af1bf2f00571"}
Jul 09 18:15:41.830850 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.830816    2572 generic.go:356] "Generic (PLEG): container finished" podID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerID="e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a" exitCode=2
Jul 09 18:15:41.830945 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.830899    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" event={"ID":"4b04426a-61f4-42e6-9c88-f6057862a1c5","Type":"ContainerDied","Data":"e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a"}
Jul 09 18:15:41.838050 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.837936    2572 scope.go:117] "RemoveContainer" containerID="152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404"
Jul 09 18:15:41.838243 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:41.838221    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404\": container with ID starting with 152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404 not found: ID does not exist" containerID="152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404"
Jul 09 18:15:41.838372 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.838344    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404"} err="failed to get container status \"152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404\": rpc error: code = NotFound desc = could not find container \"152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404\": container with ID starting with 152bf61a560bf837018e662410bad831b2cfe9e24cc6d8b095ed649220723404 not found: ID does not exist"
Jul 09 18:15:41.853994 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.853974    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"]
Jul 09 18:15:41.857388 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.857366    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7eb51-75dbc6577f-5fksf"]
Jul 09 18:15:41.866925 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.866910    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"]
Jul 09 18:15:41.869041 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:15:41.869019    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac367207_f257_4d07_9ad4_6f7f5ca22489.slice/crio-f5fe7bcf597f69f33830ebf15e0f15a326ccaffaf47f6207e11d8864ff2aa413 WatchSource:0}: Error finding container f5fe7bcf597f69f33830ebf15e0f15a326ccaffaf47f6207e11d8864ff2aa413: Status 404 returned error can't find the container with id f5fe7bcf597f69f33830ebf15e0f15a326ccaffaf47f6207e11d8864ff2aa413
Jul 09 18:15:41.934829 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:41.934803    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d36a391-f930-438b-b9a6-6297e5d30764" path="/var/lib/kubelet/pods/0d36a391-f930-438b-b9a6-6297e5d30764/volumes"
Jul 09 18:15:42.838018 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:42.837984    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" event={"ID":"ac367207-f257-4d07-9ad4-6f7f5ca22489","Type":"ContainerStarted","Data":"42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844"}
Jul 09 18:15:42.838433 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:42.838062    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:42.838433 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:42.838094    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" event={"ID":"ac367207-f257-4d07-9ad4-6f7f5ca22489","Type":"ContainerStarted","Data":"3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c"}
Jul 09 18:15:42.838433 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:42.838129    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:42.838433 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:42.838159    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" event={"ID":"ac367207-f257-4d07-9ad4-6f7f5ca22489","Type":"ContainerStarted","Data":"f5fe7bcf597f69f33830ebf15e0f15a326ccaffaf47f6207e11d8864ff2aa413"}
Jul 09 18:15:42.839540 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:42.839511    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:15:42.858747 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:42.858702    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" podStartSLOduration=1.858684574 podStartE2EDuration="1.858684574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:42.855974988 +0000 UTC m=+679.694317971" watchObservedRunningTime="2026-07-09 18:15:42.858684574 +0000 UTC m=+679.697027731"
Jul 09 18:15:43.373644 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:43.373605    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 09 18:15:43.373803 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:43.373605    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 09 18:15:43.373803 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:43.373720    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 09 18:15:43.379048 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:43.379029    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:15:43.380238 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:43.380218    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:15:43.791259 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:43.791224    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t" podUID="33474294-5f25-48ad-8fb9-13904d12dd23" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:43.843949 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:43.843913    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:15:45.854803 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.854772    2572 generic.go:356] "Generic (PLEG): container finished" podID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerID="bce7203d8879f3e58c538fdc73250b6218e333692b77bb5d38d5d5bb5c340f77" exitCode=0
Jul 09 18:15:45.855090 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.854842    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" event={"ID":"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c","Type":"ContainerDied","Data":"bce7203d8879f3e58c538fdc73250b6218e333692b77bb5d38d5d5bb5c340f77"}
Jul 09 18:15:45.855090 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.854937    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk" event={"ID":"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c","Type":"ContainerDied","Data":"419783842e0d7d8946c8e2a4d981b264d20bd4f735f94b204ba7c31edc4bd54f"}
Jul 09 18:15:45.855090 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.854958    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="419783842e0d7d8946c8e2a4d981b264d20bd4f735f94b204ba7c31edc4bd54f"
Jul 09 18:15:45.901395 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.901367    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:15:45.934700 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.934619    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kube-rbac-proxy-sar-config\") pod \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") "
Jul 09 18:15:45.934700 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.934658    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-proxy-tls\") pod \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") "
Jul 09 18:15:45.935416 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.934957    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kserve-provision-location\") pod \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") "
Jul 09 18:15:45.935416 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.934995    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-cabundle-cert\") pod \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") "
Jul 09 18:15:45.935416 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.935062    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xv67\" (UniqueName: \"kubernetes.io/projected/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kube-api-access-2xv67\") pod \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\" (UID: \"f4dc2f6c-13e3-4bac-b4f8-6acb2755397c\") "
Jul 09 18:15:45.935416 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.935146    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" (UID: "f4dc2f6c-13e3-4bac-b4f8-6acb2755397c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:15:45.935689 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.935380    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:45.935689 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.935476    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" (UID: "f4dc2f6c-13e3-4bac-b4f8-6acb2755397c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:15:45.935839 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.935723    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" (UID: "f4dc2f6c-13e3-4bac-b4f8-6acb2755397c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:15:45.938925 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.938887    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" (UID: "f4dc2f6c-13e3-4bac-b4f8-6acb2755397c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:15:45.938925 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.938908    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kube-api-access-2xv67" (OuterVolumeSpecName: "kube-api-access-2xv67") pod "f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" (UID: "f4dc2f6c-13e3-4bac-b4f8-6acb2755397c"). InnerVolumeSpecName "kube-api-access-2xv67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:15:45.993842 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:45.993819    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:15:46.036206 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.036172    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b04426a-61f4-42e6-9c88-f6057862a1c5-cabundle-cert\") pod \"4b04426a-61f4-42e6-9c88-f6057862a1c5\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") "
Jul 09 18:15:46.036349 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.036217    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b04426a-61f4-42e6-9c88-f6057862a1c5-kserve-provision-location\") pod \"4b04426a-61f4-42e6-9c88-f6057862a1c5\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") "
Jul 09 18:15:46.036349 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.036246    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdk2x\" (UniqueName: \"kubernetes.io/projected/4b04426a-61f4-42e6-9c88-f6057862a1c5-kube-api-access-wdk2x\") pod \"4b04426a-61f4-42e6-9c88-f6057862a1c5\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") "
Jul 09 18:15:46.036349 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.036294    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b04426a-61f4-42e6-9c88-f6057862a1c5-proxy-tls\") pod \"4b04426a-61f4-42e6-9c88-f6057862a1c5\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") "
Jul 09 18:15:46.036349 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.036320    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b04426a-61f4-42e6-9c88-f6057862a1c5-kube-rbac-proxy-sar-config\") pod \"4b04426a-61f4-42e6-9c88-f6057862a1c5\" (UID: \"4b04426a-61f4-42e6-9c88-f6057862a1c5\") "
Jul 09 18:15:46.036592 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.036547    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b04426a-61f4-42e6-9c88-f6057862a1c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b04426a-61f4-42e6-9c88-f6057862a1c5" (UID: "4b04426a-61f4-42e6-9c88-f6057862a1c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:15:46.036592 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.036587    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xv67\" (UniqueName: \"kubernetes.io/projected/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kube-api-access-2xv67\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:46.036699 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.036603    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:46.036699 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.036616    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-kserve-provision-location\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:46.036699 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.036629    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c-cabundle-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:46.036699 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.036687    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b04426a-61f4-42e6-9c88-f6057862a1c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b04426a-61f4-42e6-9c88-f6057862a1c5" (UID: "4b04426a-61f4-42e6-9c88-f6057862a1c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:15:46.036899 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.036771    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b04426a-61f4-42e6-9c88-f6057862a1c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b04426a-61f4-42e6-9c88-f6057862a1c5" (UID: "4b04426a-61f4-42e6-9c88-f6057862a1c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:15:46.038385 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.038362    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b04426a-61f4-42e6-9c88-f6057862a1c5-kube-api-access-wdk2x" (OuterVolumeSpecName: "kube-api-access-wdk2x") pod "4b04426a-61f4-42e6-9c88-f6057862a1c5" (UID: "4b04426a-61f4-42e6-9c88-f6057862a1c5"). InnerVolumeSpecName "kube-api-access-wdk2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:15:46.038838 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.038819    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b04426a-61f4-42e6-9c88-f6057862a1c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b04426a-61f4-42e6-9c88-f6057862a1c5" (UID: "4b04426a-61f4-42e6-9c88-f6057862a1c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:15:46.137894 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.137849    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b04426a-61f4-42e6-9c88-f6057862a1c5-cabundle-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:46.137894 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.137891    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b04426a-61f4-42e6-9c88-f6057862a1c5-kserve-provision-location\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:46.137894 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.137902    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdk2x\" (UniqueName: \"kubernetes.io/projected/4b04426a-61f4-42e6-9c88-f6057862a1c5-kube-api-access-wdk2x\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:46.137894 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.137913    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b04426a-61f4-42e6-9c88-f6057862a1c5-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:46.138132 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.137922    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b04426a-61f4-42e6-9c88-f6057862a1c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:46.860625 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.860593    2572 generic.go:356] "Generic (PLEG): container finished" podID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerID="a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373" exitCode=0
Jul 09 18:15:46.861032 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.860652    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"
Jul 09 18:15:46.861032 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.860685    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" event={"ID":"4b04426a-61f4-42e6-9c88-f6057862a1c5","Type":"ContainerDied","Data":"a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373"}
Jul 09 18:15:46.861032 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.860745    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" event={"ID":"4b04426a-61f4-42e6-9c88-f6057862a1c5","Type":"ContainerDied","Data":"abe6e99832d52d582635b43cb761348c3123b362f7e82a3f0179a6a68c7f2066"}
Jul 09 18:15:46.861032 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.860772    2572 scope.go:117] "RemoveContainer" containerID="e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a"
Jul 09 18:15:46.861032 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.860893    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"
Jul 09 18:15:46.870229 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.870214    2572 scope.go:117] "RemoveContainer" containerID="a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373"
Jul 09 18:15:46.877546 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.877530    2572 scope.go:117] "RemoveContainer" containerID="9e723f31f697bc01ef0b94570df432be90b51dc33778e5628cafb976cb7f8386"
Jul 09 18:15:46.884626 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.884549    2572 scope.go:117] "RemoveContainer" containerID="e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a"
Jul 09 18:15:46.884853 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:46.884836    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a\": container with ID starting with e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a not found: ID does not exist" containerID="e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a"
Jul 09 18:15:46.884892 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.884862    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a"} err="failed to get container status \"e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a\": rpc error: code = NotFound desc = could not find container \"e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a\": container with ID starting with e776a5146c68dd3fd1253c8962b0a7f142ea68f04fc15fc66a8c82dc7a1ad05a not found: ID does not exist"
Jul 09 18:15:46.884892 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.884881    2572 scope.go:117] "RemoveContainer" containerID="a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373"
Jul 09 18:15:46.885142 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:46.885123    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373\": container with ID starting with a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373 not found: ID does not exist" containerID="a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373"
Jul 09 18:15:46.885186 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.885148    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373"} err="failed to get container status \"a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373\": rpc error: code = NotFound desc = could not find container \"a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373\": container with ID starting with a3ca5f81129f28d4c6ae55a593cdce21f49a361c2c0049c089d2936c13738373 not found: ID does not exist"
Jul 09 18:15:46.885186 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.885163    2572 scope.go:117] "RemoveContainer" containerID="9e723f31f697bc01ef0b94570df432be90b51dc33778e5628cafb976cb7f8386"
Jul 09 18:15:46.885364 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.885347    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"]
Jul 09 18:15:46.885407 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:46.885348    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e723f31f697bc01ef0b94570df432be90b51dc33778e5628cafb976cb7f8386\": container with ID starting with 9e723f31f697bc01ef0b94570df432be90b51dc33778e5628cafb976cb7f8386 not found: ID does not exist" containerID="9e723f31f697bc01ef0b94570df432be90b51dc33778e5628cafb976cb7f8386"
Jul 09 18:15:46.885407 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.885393    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e723f31f697bc01ef0b94570df432be90b51dc33778e5628cafb976cb7f8386"} err="failed to get container status \"9e723f31f697bc01ef0b94570df432be90b51dc33778e5628cafb976cb7f8386\": rpc error: code = NotFound desc = could not find container \"9e723f31f697bc01ef0b94570df432be90b51dc33778e5628cafb976cb7f8386\": container with ID starting with 9e723f31f697bc01ef0b94570df432be90b51dc33778e5628cafb976cb7f8386 not found: ID does not exist"
Jul 09 18:15:46.888992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.888972    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx"]
Jul 09 18:15:46.899122 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.899101    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"]
Jul 09 18:15:46.903256 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.903236    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-mb2dk"]
Jul 09 18:15:46.939099 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:46.939075    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-b95b6754-55kpx" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: i/o timeout"
Jul 09 18:15:47.169492 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.169469    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:15:47.247392 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.247361    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kube-rbac-proxy-sar-config\") pod \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") "
Jul 09 18:15:47.247605 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.247439    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-proxy-tls\") pod \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") "
Jul 09 18:15:47.247605 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.247461    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kserve-provision-location\") pod \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") "
Jul 09 18:15:47.247605 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.247591    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-cabundle-cert\") pod \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") "
Jul 09 18:15:47.247784 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.247630    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2drqb\" (UniqueName: \"kubernetes.io/projected/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kube-api-access-2drqb\") pod \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\" (UID: \"f88647b4-dbbf-477a-a61c-eb28a0df5ffb\") "
Jul 09 18:15:47.247784 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.247728    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f88647b4-dbbf-477a-a61c-eb28a0df5ffb" (UID: "f88647b4-dbbf-477a-a61c-eb28a0df5ffb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:15:47.247885 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.247776    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f88647b4-dbbf-477a-a61c-eb28a0df5ffb" (UID: "f88647b4-dbbf-477a-a61c-eb28a0df5ffb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:15:47.247957 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.247940    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f88647b4-dbbf-477a-a61c-eb28a0df5ffb" (UID: "f88647b4-dbbf-477a-a61c-eb28a0df5ffb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:15:47.247994 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.247983    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:47.248024 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.248000    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kserve-provision-location\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:47.249652 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.249629    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f88647b4-dbbf-477a-a61c-eb28a0df5ffb" (UID: "f88647b4-dbbf-477a-a61c-eb28a0df5ffb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:15:47.249723 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.249647    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kube-api-access-2drqb" (OuterVolumeSpecName: "kube-api-access-2drqb") pod "f88647b4-dbbf-477a-a61c-eb28a0df5ffb" (UID: "f88647b4-dbbf-477a-a61c-eb28a0df5ffb"). InnerVolumeSpecName "kube-api-access-2drqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:15:47.349381 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.349347    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-cabundle-cert\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:47.349381 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.349379    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2drqb\" (UniqueName: \"kubernetes.io/projected/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-kube-api-access-2drqb\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:47.349598 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.349392    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88647b4-dbbf-477a-a61c-eb28a0df5ffb-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:15:47.866798 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.866766    2572 generic.go:356] "Generic (PLEG): container finished" podID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerID="65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16" exitCode=0
Jul 09 18:15:47.867267 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.866833    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"
Jul 09 18:15:47.867267 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.866858    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" event={"ID":"f88647b4-dbbf-477a-a61c-eb28a0df5ffb","Type":"ContainerDied","Data":"65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16"}
Jul 09 18:15:47.867267 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.866926    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv" event={"ID":"f88647b4-dbbf-477a-a61c-eb28a0df5ffb","Type":"ContainerDied","Data":"18aae8c1bcd77f131ffc2070cd37a757107461c9c37a12b05cf2f94cc24c2870"}
Jul 09 18:15:47.867267 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.866961    2572 scope.go:117] "RemoveContainer" containerID="753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff"
Jul 09 18:15:47.875387 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.875363    2572 scope.go:117] "RemoveContainer" containerID="65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16"
Jul 09 18:15:47.882781 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.882766    2572 scope.go:117] "RemoveContainer" containerID="2e0c020a4723320f0c314ba061ae9f16dbf66f02eb22c9c470c928ddae5c19a7"
Jul 09 18:15:47.889592 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.889571    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"]
Jul 09 18:15:47.890023 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.890009    2572 scope.go:117] "RemoveContainer" containerID="753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff"
Jul 09 18:15:47.890285 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:47.890264    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff\": container with ID starting with 753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff not found: ID does not exist" containerID="753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff"
Jul 09 18:15:47.890334 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.890293    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff"} err="failed to get container status \"753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff\": rpc error: code = NotFound desc = could not find container \"753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff\": container with ID starting with 753e77a68e1f5c2afe3ae3c6a9d09c9fba7271da27a9e7880be985f1fce18cff not found: ID does not exist"
Jul 09 18:15:47.890334 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.890309    2572 scope.go:117] "RemoveContainer" containerID="65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16"
Jul 09 18:15:47.890527 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:47.890512    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16\": container with ID starting with 65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16 not found: ID does not exist" containerID="65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16"
Jul 09 18:15:47.890610 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.890532    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16"} err="failed to get container status \"65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16\": rpc error: code = NotFound desc = could not find container \"65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16\": container with ID starting with 65b2615b689e09cbe11a935a03af6537ff64ca188eb4e0a215c89531c1050d16 not found: ID does not exist"
Jul 09 18:15:47.890610 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.890545    2572 scope.go:117] "RemoveContainer" containerID="2e0c020a4723320f0c314ba061ae9f16dbf66f02eb22c9c470c928ddae5c19a7"
Jul 09 18:15:47.890802 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:15:47.890785    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0c020a4723320f0c314ba061ae9f16dbf66f02eb22c9c470c928ddae5c19a7\": container with ID starting with 2e0c020a4723320f0c314ba061ae9f16dbf66f02eb22c9c470c928ddae5c19a7 not found: ID does not exist" containerID="2e0c020a4723320f0c314ba061ae9f16dbf66f02eb22c9c470c928ddae5c19a7"
Jul 09 18:15:47.890839 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.890806    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0c020a4723320f0c314ba061ae9f16dbf66f02eb22c9c470c928ddae5c19a7"} err="failed to get container status \"2e0c020a4723320f0c314ba061ae9f16dbf66f02eb22c9c470c928ddae5c19a7\": rpc error: code = NotFound desc = could not find container \"2e0c020a4723320f0c314ba061ae9f16dbf66f02eb22c9c470c928ddae5c19a7\": container with ID starting with 2e0c020a4723320f0c314ba061ae9f16dbf66f02eb22c9c470c928ddae5c19a7 not found: ID does not exist"
Jul 09 18:15:47.895362 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.895342    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6cbcd4578b-st5nv"]
Jul 09 18:15:47.933395 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.933367    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" path="/var/lib/kubelet/pods/4b04426a-61f4-42e6-9c88-f6057862a1c5/volumes"
Jul 09 18:15:47.933871 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.933857    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" path="/var/lib/kubelet/pods/f4dc2f6c-13e3-4bac-b4f8-6acb2755397c/volumes"
Jul 09 18:15:47.934574 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:47.934540    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" path="/var/lib/kubelet/pods/f88647b4-dbbf-477a-a61c-eb28a0df5ffb/volumes"
Jul 09 18:15:48.791458 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:48.791421    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t" podUID="33474294-5f25-48ad-8fb9-13904d12dd23" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:48.849207 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:48.849183    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:15:48.849675 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:48.849639    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:15:50.715781 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:50.715743    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 18:15:53.791349 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:53.791311    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t" podUID="33474294-5f25-48ad-8fb9-13904d12dd23" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:53.791802 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:53.791429    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:15:58.791216 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:58.791176    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t" podUID="33474294-5f25-48ad-8fb9-13904d12dd23" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:58.849863 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:15:58.849820    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:16:00.715697 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:00.715671    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:16:03.791064 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:03.791022    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t" podUID="33474294-5f25-48ad-8fb9-13904d12dd23" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:08.791362 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:08.791318    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t" podUID="33474294-5f25-48ad-8fb9-13904d12dd23" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:08.849982 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:08.849938    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:16:11.615773 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.615737    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"]
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616060    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kube-rbac-proxy"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616071    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kube-rbac-proxy"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616083    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="storage-initializer"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616088    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="storage-initializer"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616098    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d36a391-f930-438b-b9a6-6297e5d30764" containerName="switch-graph-7eb51"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616103    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d36a391-f930-438b-b9a6-6297e5d30764" containerName="switch-graph-7eb51"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616111    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kserve-container"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616117    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kserve-container"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616126    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616131    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616138    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="storage-initializer"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616144    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="storage-initializer"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616150    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="storage-initializer"
Jul 09 18:16:11.616170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616156    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="storage-initializer"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616187    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616193    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616201    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kube-rbac-proxy"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616207    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kube-rbac-proxy"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616217    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kube-rbac-proxy"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616222    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kube-rbac-proxy"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616270    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d36a391-f930-438b-b9a6-6297e5d30764" containerName="switch-graph-7eb51"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616278    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kserve-container"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616285    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4dc2f6c-13e3-4bac-b4f8-6acb2755397c" containerName="kube-rbac-proxy"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616293    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kserve-container"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616299    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kserve-container"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616303    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f88647b4-dbbf-477a-a61c-eb28a0df5ffb" containerName="kube-rbac-proxy"
Jul 09 18:16:11.616629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.616308    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b04426a-61f4-42e6-9c88-f6057862a1c5" containerName="kube-rbac-proxy"
Jul 09 18:16:11.619101 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.619085    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:16:11.621827 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.621806    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ea8a9-serving-cert\""
Jul 09 18:16:11.621827 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.621815    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ea8a9-kube-rbac-proxy-sar-config\""
Jul 09 18:16:11.630799 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.630775    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"]
Jul 09 18:16:11.755141 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.755115    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04a8c4bf-195c-413c-b61f-1c250938d8dc-proxy-tls\") pod \"switch-graph-ea8a9-858fcbdc6b-vk5wz\" (UID: \"04a8c4bf-195c-413c-b61f-1c250938d8dc\") " pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:16:11.755267 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.755166    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04a8c4bf-195c-413c-b61f-1c250938d8dc-openshift-service-ca-bundle\") pod \"switch-graph-ea8a9-858fcbdc6b-vk5wz\" (UID: \"04a8c4bf-195c-413c-b61f-1c250938d8dc\") " pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:16:11.826720 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.826693    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:16:11.855858 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.855836    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04a8c4bf-195c-413c-b61f-1c250938d8dc-proxy-tls\") pod \"switch-graph-ea8a9-858fcbdc6b-vk5wz\" (UID: \"04a8c4bf-195c-413c-b61f-1c250938d8dc\") " pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:16:11.855991 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.855898    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04a8c4bf-195c-413c-b61f-1c250938d8dc-openshift-service-ca-bundle\") pod \"switch-graph-ea8a9-858fcbdc6b-vk5wz\" (UID: \"04a8c4bf-195c-413c-b61f-1c250938d8dc\") " pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:16:11.856764 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.856738    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04a8c4bf-195c-413c-b61f-1c250938d8dc-openshift-service-ca-bundle\") pod \"switch-graph-ea8a9-858fcbdc6b-vk5wz\" (UID: \"04a8c4bf-195c-413c-b61f-1c250938d8dc\") " pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:16:11.858229 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.858209    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04a8c4bf-195c-413c-b61f-1c250938d8dc-proxy-tls\") pod \"switch-graph-ea8a9-858fcbdc6b-vk5wz\" (UID: \"04a8c4bf-195c-413c-b61f-1c250938d8dc\") " pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:16:11.929769 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.929681    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:16:11.956372 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.956349    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33474294-5f25-48ad-8fb9-13904d12dd23-proxy-tls\") pod \"33474294-5f25-48ad-8fb9-13904d12dd23\" (UID: \"33474294-5f25-48ad-8fb9-13904d12dd23\") "
Jul 09 18:16:11.956482 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.956458    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33474294-5f25-48ad-8fb9-13904d12dd23-openshift-service-ca-bundle\") pod \"33474294-5f25-48ad-8fb9-13904d12dd23\" (UID: \"33474294-5f25-48ad-8fb9-13904d12dd23\") "
Jul 09 18:16:11.956830 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.956803    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33474294-5f25-48ad-8fb9-13904d12dd23-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "33474294-5f25-48ad-8fb9-13904d12dd23" (UID: "33474294-5f25-48ad-8fb9-13904d12dd23"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:16:11.958275 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.958254    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33474294-5f25-48ad-8fb9-13904d12dd23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "33474294-5f25-48ad-8fb9-13904d12dd23" (UID: "33474294-5f25-48ad-8fb9-13904d12dd23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:16:11.973050 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.973024    2572 generic.go:356] "Generic (PLEG): container finished" podID="33474294-5f25-48ad-8fb9-13904d12dd23" containerID="ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3" exitCode=0
Jul 09 18:16:11.973165 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.973073    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"
Jul 09 18:16:11.973165 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.973110    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t" event={"ID":"33474294-5f25-48ad-8fb9-13904d12dd23","Type":"ContainerDied","Data":"ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3"}
Jul 09 18:16:11.973165 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.973161    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t" event={"ID":"33474294-5f25-48ad-8fb9-13904d12dd23","Type":"ContainerDied","Data":"a75fb82d09dfd4dfcdc4d4843c8ea6ff610c21fc35a3a5da4bd9633159ee3ec7"}
Jul 09 18:16:11.973283 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.973184    2572 scope.go:117] "RemoveContainer" containerID="ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3"
Jul 09 18:16:11.984251 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.984209    2572 scope.go:117] "RemoveContainer" containerID="ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3"
Jul 09 18:16:11.984547 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:16:11.984524    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3\": container with ID starting with ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3 not found: ID does not exist" containerID="ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3"
Jul 09 18:16:11.984547 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.984587    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3"} err="failed to get container status \"ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3\": rpc error: code = NotFound desc = could not find container \"ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3\": container with ID starting with ad0a61ab810f5726622ff4f425d6828c486fdfbc77a0ad486cee02c8767948a3 not found: ID does not exist"
Jul 09 18:16:11.998310 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:11.998265    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"]
Jul 09 18:16:12.000501 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:12.000481    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5b4bb9966-sq87t"]
Jul 09 18:16:12.057285 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:12.057258    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33474294-5f25-48ad-8fb9-13904d12dd23-openshift-service-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:16:12.057285 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:12.057282    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33474294-5f25-48ad-8fb9-13904d12dd23-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:16:12.059937 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:12.059916    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"]
Jul 09 18:16:12.062277 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:16:12.062236    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a8c4bf_195c_413c_b61f_1c250938d8dc.slice/crio-715342a580a1d8221efa68d8c2a6e97bb0d74a347faf44af0e2a07fddba5c7c4 WatchSource:0}: Error finding container 715342a580a1d8221efa68d8c2a6e97bb0d74a347faf44af0e2a07fddba5c7c4: Status 404 returned error can't find the container with id 715342a580a1d8221efa68d8c2a6e97bb0d74a347faf44af0e2a07fddba5c7c4
Jul 09 18:16:12.979217 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:12.979174    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz" event={"ID":"04a8c4bf-195c-413c-b61f-1c250938d8dc","Type":"ContainerStarted","Data":"3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca"}
Jul 09 18:16:12.979619 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:12.979268    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz" event={"ID":"04a8c4bf-195c-413c-b61f-1c250938d8dc","Type":"ContainerStarted","Data":"715342a580a1d8221efa68d8c2a6e97bb0d74a347faf44af0e2a07fddba5c7c4"}
Jul 09 18:16:12.979619 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:12.979297    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:16:13.000220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:13.000174    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz" podStartSLOduration=2.000160759 podStartE2EDuration="2.000160759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:12.997673435 +0000 UTC m=+709.836016418" watchObservedRunningTime="2026-07-09 18:16:13.000160759 +0000 UTC m=+709.838503744"
Jul 09 18:16:13.932062 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:13.932031    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33474294-5f25-48ad-8fb9-13904d12dd23" path="/var/lib/kubelet/pods/33474294-5f25-48ad-8fb9-13904d12dd23/volumes"
Jul 09 18:16:18.849902 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:18.849864    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:16:18.989781 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:18.989752    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:16:28.850796 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:28.850767    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:16:41.402049 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.402013    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"]
Jul 09 18:16:41.402518 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.402368    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33474294-5f25-48ad-8fb9-13904d12dd23" containerName="model-chainer"
Jul 09 18:16:41.402518 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.402382    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="33474294-5f25-48ad-8fb9-13904d12dd23" containerName="model-chainer"
Jul 09 18:16:41.402518 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.402451    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="33474294-5f25-48ad-8fb9-13904d12dd23" containerName="model-chainer"
Jul 09 18:16:41.405693 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.405674    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:16:41.408017 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.407997    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-04bd4-kube-rbac-proxy-sar-config\""
Jul 09 18:16:41.408105 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.408001    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-04bd4-serving-cert\""
Jul 09 18:16:41.416114 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.416088    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"]
Jul 09 18:16:41.502413 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.502380    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1-openshift-service-ca-bundle\") pod \"sequence-graph-04bd4-7dcfc7455-bmt9g\" (UID: \"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1\") " pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:16:41.502628 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.502442    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1-proxy-tls\") pod \"sequence-graph-04bd4-7dcfc7455-bmt9g\" (UID: \"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1\") " pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:16:41.603230 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.603194    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1-proxy-tls\") pod \"sequence-graph-04bd4-7dcfc7455-bmt9g\" (UID: \"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1\") " pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:16:41.603401 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.603262    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1-openshift-service-ca-bundle\") pod \"sequence-graph-04bd4-7dcfc7455-bmt9g\" (UID: \"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1\") " pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:16:41.603844 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.603827    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1-openshift-service-ca-bundle\") pod \"sequence-graph-04bd4-7dcfc7455-bmt9g\" (UID: \"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1\") " pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:16:41.605643 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.605609    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1-proxy-tls\") pod \"sequence-graph-04bd4-7dcfc7455-bmt9g\" (UID: \"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1\") " pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:16:41.716480 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.716446    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:16:41.852743 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:41.852710    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"]
Jul 09 18:16:41.855948 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:16:41.855918    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8da6f1b_b79a_4e30_be51_e3cd5d6e64f1.slice/crio-a8b345ba49dfcd7b1662f51146ac29d6c21640022ed09809e26288101ce9b1bf WatchSource:0}: Error finding container a8b345ba49dfcd7b1662f51146ac29d6c21640022ed09809e26288101ce9b1bf: Status 404 returned error can't find the container with id a8b345ba49dfcd7b1662f51146ac29d6c21640022ed09809e26288101ce9b1bf
Jul 09 18:16:42.101315 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:42.101228    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g" event={"ID":"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1","Type":"ContainerStarted","Data":"1651841b302bb4c131fc302846b16a879d0d617e7e364826bd163dcfcc5d52fb"}
Jul 09 18:16:42.101315 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:42.101275    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g" event={"ID":"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1","Type":"ContainerStarted","Data":"a8b345ba49dfcd7b1662f51146ac29d6c21640022ed09809e26288101ce9b1bf"}
Jul 09 18:16:42.101518 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:42.101354    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:16:42.118952 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:42.118900    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g" podStartSLOduration=1.118885826 podStartE2EDuration="1.118885826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:42.118273525 +0000 UTC m=+738.956616508" watchObservedRunningTime="2026-07-09 18:16:42.118885826 +0000 UTC m=+738.957228809"
Jul 09 18:16:48.110969 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:16:48.110938    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:19:23.703068 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:19:23.703040    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:19:23.704496 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:19:23.704479    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:19:23.748089 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:19:23.748064    2572 scope.go:117] "RemoveContainer" containerID="5e9decdbf1d2931ad857ad8b31631666ab33594373d0638d2d2cdfb04e435d7c"
Jul 09 18:20:23.761476 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:20:23.761440    2572 scope.go:117] "RemoveContainer" containerID="b86a08d2abfea72e02df52815750eb0fc14f5cddddc84c824a54af1bf2f00571"
Jul 09 18:20:23.769961 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:20:23.769937    2572 scope.go:117] "RemoveContainer" containerID="bce7203d8879f3e58c538fdc73250b6218e333692b77bb5d38d5d5bb5c340f77"
Jul 09 18:24:23.729459 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:23.729433    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:24:23.731385 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:23.731362    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:24:26.151824 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.151793    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"]
Jul 09 18:24:26.152204 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.152032    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz" podUID="04a8c4bf-195c-413c-b61f-1c250938d8dc" containerName="switch-graph-ea8a9" containerID="cri-o://3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca" gracePeriod=30
Jul 09 18:24:26.305989 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.305958    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"]
Jul 09 18:24:26.306201 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.306174    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kserve-container" containerID="cri-o://90c9561b2fa832a0f84871c147d329b48a9b82756a1082e6647b29e5fe0d192a" gracePeriod=30
Jul 09 18:24:26.306296 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.306200    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kube-rbac-proxy" containerID="cri-o://9571edb4d15bac7b869e322c39e88d1fae3c4702fe52045943fef6e7ae74d207" gracePeriod=30
Jul 09 18:24:26.392321 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.392286    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"]
Jul 09 18:24:26.396316 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.396297    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.398780 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.398758    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d3ac8-predictor-serving-cert\""
Jul 09 18:24:26.398888 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.398824    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d3ac8-kube-rbac-proxy-sar-config\""
Jul 09 18:24:26.408955 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.408932    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"]
Jul 09 18:24:26.450690 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.450666    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9m9w\" (UniqueName: \"kubernetes.io/projected/3beb597f-6558-466c-9da9-4fc0286584b4-kube-api-access-k9m9w\") pod \"success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf\" (UID: \"3beb597f-6558-466c-9da9-4fc0286584b4\") " pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.450794 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.450718    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3beb597f-6558-466c-9da9-4fc0286584b4-proxy-tls\") pod \"success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf\" (UID: \"3beb597f-6558-466c-9da9-4fc0286584b4\") " pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.450794 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.450788    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3beb597f-6558-466c-9da9-4fc0286584b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf\" (UID: \"3beb597f-6558-466c-9da9-4fc0286584b4\") " pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.551767 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.551743    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3beb597f-6558-466c-9da9-4fc0286584b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf\" (UID: \"3beb597f-6558-466c-9da9-4fc0286584b4\") " pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.551852 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.551787    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9m9w\" (UniqueName: \"kubernetes.io/projected/3beb597f-6558-466c-9da9-4fc0286584b4-kube-api-access-k9m9w\") pod \"success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf\" (UID: \"3beb597f-6558-466c-9da9-4fc0286584b4\") " pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.551852 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.551839    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3beb597f-6558-466c-9da9-4fc0286584b4-proxy-tls\") pod \"success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf\" (UID: \"3beb597f-6558-466c-9da9-4fc0286584b4\") " pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.552359 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.552340    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3beb597f-6558-466c-9da9-4fc0286584b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf\" (UID: \"3beb597f-6558-466c-9da9-4fc0286584b4\") " pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.554096 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.554075    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3beb597f-6558-466c-9da9-4fc0286584b4-proxy-tls\") pod \"success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf\" (UID: \"3beb597f-6558-466c-9da9-4fc0286584b4\") " pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.562548 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.562518    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9m9w\" (UniqueName: \"kubernetes.io/projected/3beb597f-6558-466c-9da9-4fc0286584b4-kube-api-access-k9m9w\") pod \"success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf\" (UID: \"3beb597f-6558-466c-9da9-4fc0286584b4\") " pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.707999 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.707977    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.853000 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.852978    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"]
Jul 09 18:24:26.855486 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:24:26.855464    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3beb597f_6558_466c_9da9_4fc0286584b4.slice/crio-ba9aa562908eccbc432b13e19b709882ba95bc0ea60f55af096c2bb04449c971 WatchSource:0}: Error finding container ba9aa562908eccbc432b13e19b709882ba95bc0ea60f55af096c2bb04449c971: Status 404 returned error can't find the container with id ba9aa562908eccbc432b13e19b709882ba95bc0ea60f55af096c2bb04449c971
Jul 09 18:24:26.968244 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.968220    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" event={"ID":"3beb597f-6558-466c-9da9-4fc0286584b4","Type":"ContainerStarted","Data":"7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c"}
Jul 09 18:24:26.968344 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.968272    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" event={"ID":"3beb597f-6558-466c-9da9-4fc0286584b4","Type":"ContainerStarted","Data":"0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750"}
Jul 09 18:24:26.968344 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.968298    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" event={"ID":"3beb597f-6558-466c-9da9-4fc0286584b4","Type":"ContainerStarted","Data":"ba9aa562908eccbc432b13e19b709882ba95bc0ea60f55af096c2bb04449c971"}
Jul 09 18:24:26.968344 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.968325    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.968512 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.968350    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:26.969405 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.969376    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 09 18:24:26.970288 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.970269    2572 generic.go:356] "Generic (PLEG): container finished" podID="98b183e2-8018-400f-adca-4853efd8cd70" containerID="9571edb4d15bac7b869e322c39e88d1fae3c4702fe52045943fef6e7ae74d207" exitCode=2
Jul 09 18:24:26.970364 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.970330    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" event={"ID":"98b183e2-8018-400f-adca-4853efd8cd70","Type":"ContainerDied","Data":"9571edb4d15bac7b869e322c39e88d1fae3c4702fe52045943fef6e7ae74d207"}
Jul 09 18:24:26.996317 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:26.996278    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" podStartSLOduration=0.996263798 podStartE2EDuration="996.263798ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:26.995019641 +0000 UTC m=+1203.833362628" watchObservedRunningTime="2026-07-09 18:24:26.996263798 +0000 UTC m=+1203.834606783"
Jul 09 18:24:27.974358 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:27.974324    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 09 18:24:28.981823 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:28.981795    2572 generic.go:356] "Generic (PLEG): container finished" podID="98b183e2-8018-400f-adca-4853efd8cd70" containerID="90c9561b2fa832a0f84871c147d329b48a9b82756a1082e6647b29e5fe0d192a" exitCode=0
Jul 09 18:24:28.982195 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:28.981826    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" event={"ID":"98b183e2-8018-400f-adca-4853efd8cd70","Type":"ContainerDied","Data":"90c9561b2fa832a0f84871c147d329b48a9b82756a1082e6647b29e5fe0d192a"}
Jul 09 18:24:28.988469 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:28.988444    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz" podUID="04a8c4bf-195c-413c-b61f-1c250938d8dc" containerName="switch-graph-ea8a9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:24:29.054072 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.054052    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:24:29.177138 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.177075    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8789p\" (UniqueName: \"kubernetes.io/projected/98b183e2-8018-400f-adca-4853efd8cd70-kube-api-access-8789p\") pod \"98b183e2-8018-400f-adca-4853efd8cd70\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") "
Jul 09 18:24:29.177138 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.177118    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98b183e2-8018-400f-adca-4853efd8cd70-proxy-tls\") pod \"98b183e2-8018-400f-adca-4853efd8cd70\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") "
Jul 09 18:24:29.177414 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.177213    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98b183e2-8018-400f-adca-4853efd8cd70-kube-rbac-proxy-sar-config\") pod \"98b183e2-8018-400f-adca-4853efd8cd70\" (UID: \"98b183e2-8018-400f-adca-4853efd8cd70\") "
Jul 09 18:24:29.177589 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.177544    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b183e2-8018-400f-adca-4853efd8cd70-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98b183e2-8018-400f-adca-4853efd8cd70" (UID: "98b183e2-8018-400f-adca-4853efd8cd70"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:29.179216 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.179190    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b183e2-8018-400f-adca-4853efd8cd70-kube-api-access-8789p" (OuterVolumeSpecName: "kube-api-access-8789p") pod "98b183e2-8018-400f-adca-4853efd8cd70" (UID: "98b183e2-8018-400f-adca-4853efd8cd70"). InnerVolumeSpecName "kube-api-access-8789p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:29.179604 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.179439    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b183e2-8018-400f-adca-4853efd8cd70-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98b183e2-8018-400f-adca-4853efd8cd70" (UID: "98b183e2-8018-400f-adca-4853efd8cd70"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:29.278458 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.278435    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8789p\" (UniqueName: \"kubernetes.io/projected/98b183e2-8018-400f-adca-4853efd8cd70-kube-api-access-8789p\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:24:29.278458 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.278456    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98b183e2-8018-400f-adca-4853efd8cd70-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:24:29.278600 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.278467    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98b183e2-8018-400f-adca-4853efd8cd70-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:24:29.987531 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.987495    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk" event={"ID":"98b183e2-8018-400f-adca-4853efd8cd70","Type":"ContainerDied","Data":"990e552990c186f9f0d1996e60e34f18d01e30c48e74c07e0b6c3e678248d22e"}
Jul 09 18:24:29.987531 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.987522    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"
Jul 09 18:24:29.987981 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.987583    2572 scope.go:117] "RemoveContainer" containerID="9571edb4d15bac7b869e322c39e88d1fae3c4702fe52045943fef6e7ae74d207"
Jul 09 18:24:29.995407 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:29.995393    2572 scope.go:117] "RemoveContainer" containerID="90c9561b2fa832a0f84871c147d329b48a9b82756a1082e6647b29e5fe0d192a"
Jul 09 18:24:30.004791 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:30.004762    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"]
Jul 09 18:24:30.007367 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:30.007348    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ea8a9-predictor-7487c75d4d-trjdk"]
Jul 09 18:24:31.932955 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:31.932923    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98b183e2-8018-400f-adca-4853efd8cd70" path="/var/lib/kubelet/pods/98b183e2-8018-400f-adca-4853efd8cd70/volumes"
Jul 09 18:24:32.978710 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:32.978681    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:24:32.979143 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:32.979120    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 09 18:24:33.987152 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:33.987119    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz" podUID="04a8c4bf-195c-413c-b61f-1c250938d8dc" containerName="switch-graph-ea8a9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:24:38.986900 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:38.986857    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz" podUID="04a8c4bf-195c-413c-b61f-1c250938d8dc" containerName="switch-graph-ea8a9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:24:38.987345 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:38.986965    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:24:42.979131 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:42.979089    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 09 18:24:43.987426 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:43.987384    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz" podUID="04a8c4bf-195c-413c-b61f-1c250938d8dc" containerName="switch-graph-ea8a9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:24:48.986792 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:48.986748    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz" podUID="04a8c4bf-195c-413c-b61f-1c250938d8dc" containerName="switch-graph-ea8a9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:24:52.979628 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:52.979592    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 09 18:24:53.987189 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:53.987156    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz" podUID="04a8c4bf-195c-413c-b61f-1c250938d8dc" containerName="switch-graph-ea8a9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:24:55.983688 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:55.983659    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"]
Jul 09 18:24:55.984362 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:55.983834    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g" podUID="b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" containerName="sequence-graph-04bd4" containerID="cri-o://1651841b302bb4c131fc302846b16a879d0d617e7e364826bd163dcfcc5d52fb" gracePeriod=30
Jul 09 18:24:56.115595 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.115544    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"]
Jul 09 18:24:56.115810 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.115780    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kserve-container" containerID="cri-o://3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c" gracePeriod=30
Jul 09 18:24:56.115942 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.115818    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kube-rbac-proxy" containerID="cri-o://42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844" gracePeriod=30
Jul 09 18:24:56.178367 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.178347    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"]
Jul 09 18:24:56.178691 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.178678    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kube-rbac-proxy"
Jul 09 18:24:56.178762 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.178693    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kube-rbac-proxy"
Jul 09 18:24:56.178762 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.178707    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kserve-container"
Jul 09 18:24:56.178762 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.178713    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kserve-container"
Jul 09 18:24:56.178875 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.178763    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kserve-container"
Jul 09 18:24:56.178875 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.178772    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b183e2-8018-400f-adca-4853efd8cd70" containerName="kube-rbac-proxy"
Jul 09 18:24:56.181664 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.181644    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:56.184358 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.184339    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2e68f-predictor-serving-cert\""
Jul 09 18:24:56.184477 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.184426    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2e68f-kube-rbac-proxy-sar-config\""
Jul 09 18:24:56.192259 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.192242    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"]
Jul 09 18:24:56.276882 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.276855    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f5340ec-c5f9-4511-aa76-c7faf296b184-proxy-tls\") pod \"success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj\" (UID: \"5f5340ec-c5f9-4511-aa76-c7faf296b184\") " pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:56.277036 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.276903    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czkxg\" (UniqueName: \"kubernetes.io/projected/5f5340ec-c5f9-4511-aa76-c7faf296b184-kube-api-access-czkxg\") pod \"success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj\" (UID: \"5f5340ec-c5f9-4511-aa76-c7faf296b184\") " pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:56.277036 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.276924    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f5340ec-c5f9-4511-aa76-c7faf296b184-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj\" (UID: \"5f5340ec-c5f9-4511-aa76-c7faf296b184\") " pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:56.364499 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.364477    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:24:56.377680 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.377656    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkxg\" (UniqueName: \"kubernetes.io/projected/5f5340ec-c5f9-4511-aa76-c7faf296b184-kube-api-access-czkxg\") pod \"success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj\" (UID: \"5f5340ec-c5f9-4511-aa76-c7faf296b184\") " pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:56.377782 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.377687    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f5340ec-c5f9-4511-aa76-c7faf296b184-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj\" (UID: \"5f5340ec-c5f9-4511-aa76-c7faf296b184\") " pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:56.377782 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.377747    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f5340ec-c5f9-4511-aa76-c7faf296b184-proxy-tls\") pod \"success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj\" (UID: \"5f5340ec-c5f9-4511-aa76-c7faf296b184\") " pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:56.378303 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.378282    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f5340ec-c5f9-4511-aa76-c7faf296b184-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj\" (UID: \"5f5340ec-c5f9-4511-aa76-c7faf296b184\") " pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:56.380115 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.380092    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f5340ec-c5f9-4511-aa76-c7faf296b184-proxy-tls\") pod \"success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj\" (UID: \"5f5340ec-c5f9-4511-aa76-c7faf296b184\") " pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:56.386101 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.386015    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czkxg\" (UniqueName: \"kubernetes.io/projected/5f5340ec-c5f9-4511-aa76-c7faf296b184-kube-api-access-czkxg\") pod \"success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj\" (UID: \"5f5340ec-c5f9-4511-aa76-c7faf296b184\") " pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:56.478207 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.478188    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04a8c4bf-195c-413c-b61f-1c250938d8dc-proxy-tls\") pod \"04a8c4bf-195c-413c-b61f-1c250938d8dc\" (UID: \"04a8c4bf-195c-413c-b61f-1c250938d8dc\") "
Jul 09 18:24:56.478317 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.478219    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04a8c4bf-195c-413c-b61f-1c250938d8dc-openshift-service-ca-bundle\") pod \"04a8c4bf-195c-413c-b61f-1c250938d8dc\" (UID: \"04a8c4bf-195c-413c-b61f-1c250938d8dc\") "
Jul 09 18:24:56.478633 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.478608    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a8c4bf-195c-413c-b61f-1c250938d8dc-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "04a8c4bf-195c-413c-b61f-1c250938d8dc" (UID: "04a8c4bf-195c-413c-b61f-1c250938d8dc"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:56.480013 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.479996    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a8c4bf-195c-413c-b61f-1c250938d8dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04a8c4bf-195c-413c-b61f-1c250938d8dc" (UID: "04a8c4bf-195c-413c-b61f-1c250938d8dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:56.520336 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.520282    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:56.579913 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.579875    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04a8c4bf-195c-413c-b61f-1c250938d8dc-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:24:56.580042 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.579917    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04a8c4bf-195c-413c-b61f-1c250938d8dc-openshift-service-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:24:56.646389 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:56.646296    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"]
Jul 09 18:24:56.648527 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:24:56.648504    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5340ec_c5f9_4511_aa76_c7faf296b184.slice/crio-62d0c3c7d6742e0b8816f95a20c04fa832df0a6760d47f028227747d1f261661 WatchSource:0}: Error finding container 62d0c3c7d6742e0b8816f95a20c04fa832df0a6760d47f028227747d1f261661: Status 404 returned error can't find the container with id 62d0c3c7d6742e0b8816f95a20c04fa832df0a6760d47f028227747d1f261661
Jul 09 18:24:57.094859 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.094826    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" event={"ID":"5f5340ec-c5f9-4511-aa76-c7faf296b184","Type":"ContainerStarted","Data":"6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8"}
Jul 09 18:24:57.095305 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.094893    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:57.095305 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.094923    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" event={"ID":"5f5340ec-c5f9-4511-aa76-c7faf296b184","Type":"ContainerStarted","Data":"d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557"}
Jul 09 18:24:57.095305 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.094953    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:24:57.095305 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.094979    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" event={"ID":"5f5340ec-c5f9-4511-aa76-c7faf296b184","Type":"ContainerStarted","Data":"62d0c3c7d6742e0b8816f95a20c04fa832df0a6760d47f028227747d1f261661"}
Jul 09 18:24:57.096211 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.096185    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 09 18:24:57.096733 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.096710    2572 generic.go:356] "Generic (PLEG): container finished" podID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerID="42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844" exitCode=2
Jul 09 18:24:57.096819 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.096780    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" event={"ID":"ac367207-f257-4d07-9ad4-6f7f5ca22489","Type":"ContainerDied","Data":"42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844"}
Jul 09 18:24:57.097865 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.097845    2572 generic.go:356] "Generic (PLEG): container finished" podID="04a8c4bf-195c-413c-b61f-1c250938d8dc" containerID="3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca" exitCode=0
Jul 09 18:24:57.097950 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.097882    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz" event={"ID":"04a8c4bf-195c-413c-b61f-1c250938d8dc","Type":"ContainerDied","Data":"3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca"}
Jul 09 18:24:57.097950 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.097882    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"
Jul 09 18:24:57.097950 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.097920    2572 scope.go:117] "RemoveContainer" containerID="3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca"
Jul 09 18:24:57.098050 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.097910    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz" event={"ID":"04a8c4bf-195c-413c-b61f-1c250938d8dc","Type":"ContainerDied","Data":"715342a580a1d8221efa68d8c2a6e97bb0d74a347faf44af0e2a07fddba5c7c4"}
Jul 09 18:24:57.105882 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.105869    2572 scope.go:117] "RemoveContainer" containerID="3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca"
Jul 09 18:24:57.106104 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:24:57.106088    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca\": container with ID starting with 3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca not found: ID does not exist" containerID="3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca"
Jul 09 18:24:57.106150 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.106110    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca"} err="failed to get container status \"3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca\": rpc error: code = NotFound desc = could not find container \"3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca\": container with ID starting with 3a53db9c79d6f1a9071036d53ca21f5dca0be4cf30d036bceb2b2b0db3c097ca not found: ID does not exist"
Jul 09 18:24:57.172561 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.172519    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" podStartSLOduration=1.172507335 podStartE2EDuration="1.172507335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:57.143259512 +0000 UTC m=+1233.981602495" watchObservedRunningTime="2026-07-09 18:24:57.172507335 +0000 UTC m=+1234.010850323"
Jul 09 18:24:57.173178 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.173164    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"]
Jul 09 18:24:57.185233 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.185215    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ea8a9-858fcbdc6b-vk5wz"]
Jul 09 18:24:57.936801 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:57.936770    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04a8c4bf-195c-413c-b61f-1c250938d8dc" path="/var/lib/kubelet/pods/04a8c4bf-195c-413c-b61f-1c250938d8dc/volumes"
Jul 09 18:24:58.102260 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:58.102224    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 09 18:24:58.108906 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:58.108877    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g" podUID="b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" containerName="sequence-graph-04bd4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:24:58.844215 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:58.844181    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 09 18:24:58.850637 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:58.850608    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:24:58.957450 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:58.957429    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:24:59.099533 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.099462    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac367207-f257-4d07-9ad4-6f7f5ca22489-kube-rbac-proxy-sar-config\") pod \"ac367207-f257-4d07-9ad4-6f7f5ca22489\" (UID: \"ac367207-f257-4d07-9ad4-6f7f5ca22489\") "
Jul 09 18:24:59.099679 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.099534    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac367207-f257-4d07-9ad4-6f7f5ca22489-proxy-tls\") pod \"ac367207-f257-4d07-9ad4-6f7f5ca22489\" (UID: \"ac367207-f257-4d07-9ad4-6f7f5ca22489\") "
Jul 09 18:24:59.099679 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.099609    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhkd7\" (UniqueName: \"kubernetes.io/projected/ac367207-f257-4d07-9ad4-6f7f5ca22489-kube-api-access-rhkd7\") pod \"ac367207-f257-4d07-9ad4-6f7f5ca22489\" (UID: \"ac367207-f257-4d07-9ad4-6f7f5ca22489\") "
Jul 09 18:24:59.099905 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.099871    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac367207-f257-4d07-9ad4-6f7f5ca22489-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ac367207-f257-4d07-9ad4-6f7f5ca22489" (UID: "ac367207-f257-4d07-9ad4-6f7f5ca22489"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:59.101504 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.101480    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac367207-f257-4d07-9ad4-6f7f5ca22489-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ac367207-f257-4d07-9ad4-6f7f5ca22489" (UID: "ac367207-f257-4d07-9ad4-6f7f5ca22489"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:59.101504 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.101494    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac367207-f257-4d07-9ad4-6f7f5ca22489-kube-api-access-rhkd7" (OuterVolumeSpecName: "kube-api-access-rhkd7") pod "ac367207-f257-4d07-9ad4-6f7f5ca22489" (UID: "ac367207-f257-4d07-9ad4-6f7f5ca22489"). InnerVolumeSpecName "kube-api-access-rhkd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:59.107525 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.107500    2572 generic.go:356] "Generic (PLEG): container finished" podID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerID="3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c" exitCode=0
Jul 09 18:24:59.107895 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.107545    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" event={"ID":"ac367207-f257-4d07-9ad4-6f7f5ca22489","Type":"ContainerDied","Data":"3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c"}
Jul 09 18:24:59.107895 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.107622    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw" event={"ID":"ac367207-f257-4d07-9ad4-6f7f5ca22489","Type":"ContainerDied","Data":"f5fe7bcf597f69f33830ebf15e0f15a326ccaffaf47f6207e11d8864ff2aa413"}
Jul 09 18:24:59.107895 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.107657    2572 scope.go:117] "RemoveContainer" containerID="42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844"
Jul 09 18:24:59.107895 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.107660    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"
Jul 09 18:24:59.116403 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.116381    2572 scope.go:117] "RemoveContainer" containerID="3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c"
Jul 09 18:24:59.123898 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.123866    2572 scope.go:117] "RemoveContainer" containerID="42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844"
Jul 09 18:24:59.124131 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:24:59.124114    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844\": container with ID starting with 42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844 not found: ID does not exist" containerID="42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844"
Jul 09 18:24:59.124190 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.124137    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844"} err="failed to get container status \"42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844\": rpc error: code = NotFound desc = could not find container \"42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844\": container with ID starting with 42afde9f87fc5d478638908a43f08ff71cee685379c963635c7bdb395c9fa844 not found: ID does not exist"
Jul 09 18:24:59.124190 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.124155    2572 scope.go:117] "RemoveContainer" containerID="3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c"
Jul 09 18:24:59.124378 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:24:59.124363    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c\": container with ID starting with 3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c not found: ID does not exist" containerID="3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c"
Jul 09 18:24:59.124419 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.124383    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c"} err="failed to get container status \"3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c\": rpc error: code = NotFound desc = could not find container \"3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c\": container with ID starting with 3c3b99449b018bbb2035958a8f6fd81453a1d7b6971c01c4215dbbda2e6cfb7c not found: ID does not exist"
Jul 09 18:24:59.131432 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.131407    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"]
Jul 09 18:24:59.135873 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.135853    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-04bd4-predictor-5449bcfd5b-5l4kw"]
Jul 09 18:24:59.200875 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.200856    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac367207-f257-4d07-9ad4-6f7f5ca22489-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:24:59.200875 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.200874    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhkd7\" (UniqueName: \"kubernetes.io/projected/ac367207-f257-4d07-9ad4-6f7f5ca22489-kube-api-access-rhkd7\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:24:59.200984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.200885    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac367207-f257-4d07-9ad4-6f7f5ca22489-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:24:59.933966 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:24:59.933940    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" path="/var/lib/kubelet/pods/ac367207-f257-4d07-9ad4-6f7f5ca22489/volumes"
Jul 09 18:25:02.979386 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:02.979350    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 09 18:25:03.107374 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:03.107348    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:25:03.107840 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:03.107815    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 09 18:25:03.109431 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:03.109412    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g" podUID="b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" containerName="sequence-graph-04bd4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:25:08.108880 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:08.108846    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g" podUID="b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" containerName="sequence-graph-04bd4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:25:08.109247 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:08.108943    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:25:12.980161 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:12.980131    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:25:13.108758 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:13.108718    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 09 18:25:13.108992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:13.108967    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g" podUID="b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" containerName="sequence-graph-04bd4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:25:18.108923 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:18.108888    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g" podUID="b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" containerName="sequence-graph-04bd4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:25:23.108780 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:23.108735    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 09 18:25:23.109877 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:23.109841    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g" podUID="b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" containerName="sequence-graph-04bd4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:25:26.217600 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.217542    2572 generic.go:356] "Generic (PLEG): container finished" podID="b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" containerID="1651841b302bb4c131fc302846b16a879d0d617e7e364826bd163dcfcc5d52fb" exitCode=0
Jul 09 18:25:26.217899 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.217621    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g" event={"ID":"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1","Type":"ContainerDied","Data":"1651841b302bb4c131fc302846b16a879d0d617e7e364826bd163dcfcc5d52fb"}
Jul 09 18:25:26.623058 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.623037    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:25:26.637825 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.637803    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"]
Jul 09 18:25:26.638118 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.638106    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kserve-container"
Jul 09 18:25:26.638162 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.638120    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kserve-container"
Jul 09 18:25:26.638162 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.638135    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kube-rbac-proxy"
Jul 09 18:25:26.638162 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.638143    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kube-rbac-proxy"
Jul 09 18:25:26.638254 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.638164    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" containerName="sequence-graph-04bd4"
Jul 09 18:25:26.638254 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.638171    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" containerName="sequence-graph-04bd4"
Jul 09 18:25:26.638254 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.638178    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a8c4bf-195c-413c-b61f-1c250938d8dc" containerName="switch-graph-ea8a9"
Jul 09 18:25:26.638254 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.638183    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a8c4bf-195c-413c-b61f-1c250938d8dc" containerName="switch-graph-ea8a9"
Jul 09 18:25:26.638254 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.638235    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kube-rbac-proxy"
Jul 09 18:25:26.638254 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.638243    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a8c4bf-195c-413c-b61f-1c250938d8dc" containerName="switch-graph-ea8a9"
Jul 09 18:25:26.638254 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.638249    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" containerName="sequence-graph-04bd4"
Jul 09 18:25:26.638453 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.638258    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac367207-f257-4d07-9ad4-6f7f5ca22489" containerName="kserve-container"
Jul 09 18:25:26.641992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.641969    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:25:26.647159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.647144    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d3ac8-serving-cert\""
Jul 09 18:25:26.647565 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.647535    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d3ac8-kube-rbac-proxy-sar-config\""
Jul 09 18:25:26.687205 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.687177    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"]
Jul 09 18:25:26.699218 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.699199    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1-openshift-service-ca-bundle\") pod \"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1\" (UID: \"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1\") "
Jul 09 18:25:26.699297 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.699247    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1-proxy-tls\") pod \"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1\" (UID: \"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1\") "
Jul 09 18:25:26.699405 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.699388    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a-proxy-tls\") pod \"ensemble-graph-d3ac8-74c8fdd6dd-xmccm\" (UID: \"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a\") " pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:25:26.699455 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.699438    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a-openshift-service-ca-bundle\") pod \"ensemble-graph-d3ac8-74c8fdd6dd-xmccm\" (UID: \"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a\") " pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:25:26.699595 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.699572    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" (UID: "b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:25:26.701284 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.701257    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" (UID: "b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:25:26.800537 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.800485    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a-proxy-tls\") pod \"ensemble-graph-d3ac8-74c8fdd6dd-xmccm\" (UID: \"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a\") " pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:25:26.800537 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.800517    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a-openshift-service-ca-bundle\") pod \"ensemble-graph-d3ac8-74c8fdd6dd-xmccm\" (UID: \"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a\") " pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:25:26.800670 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.800570    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1-openshift-service-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:25:26.800670 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.800586    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:25:26.801129 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.801107    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a-openshift-service-ca-bundle\") pod \"ensemble-graph-d3ac8-74c8fdd6dd-xmccm\" (UID: \"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a\") " pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:25:26.802708 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.802687    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a-proxy-tls\") pod \"ensemble-graph-d3ac8-74c8fdd6dd-xmccm\" (UID: \"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a\") " pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:25:26.952863 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:26.952838    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:25:27.083769 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:27.083745    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"]
Jul 09 18:25:27.085651 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:25:27.085619    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6d7c47_90d5_4c71_8cdb_80cf301e9b2a.slice/crio-fc40ce4dae16248dd5125962c5c697670885f0beaff798700d0299d0731c610b WatchSource:0}: Error finding container fc40ce4dae16248dd5125962c5c697670885f0beaff798700d0299d0731c610b: Status 404 returned error can't find the container with id fc40ce4dae16248dd5125962c5c697670885f0beaff798700d0299d0731c610b
Jul 09 18:25:27.222664 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:27.222628    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm" event={"ID":"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a","Type":"ContainerStarted","Data":"a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51"}
Jul 09 18:25:27.223022 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:27.222680    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm" event={"ID":"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a","Type":"ContainerStarted","Data":"fc40ce4dae16248dd5125962c5c697670885f0beaff798700d0299d0731c610b"}
Jul 09 18:25:27.223022 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:27.222706    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:25:27.223925 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:27.223907    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g" event={"ID":"b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1","Type":"ContainerDied","Data":"a8b345ba49dfcd7b1662f51146ac29d6c21640022ed09809e26288101ce9b1bf"}
Jul 09 18:25:27.224003 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:27.223945    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"
Jul 09 18:25:27.224044 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:27.223950    2572 scope.go:117] "RemoveContainer" containerID="1651841b302bb4c131fc302846b16a879d0d617e7e364826bd163dcfcc5d52fb"
Jul 09 18:25:27.242791 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:27.242751    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm" podStartSLOduration=1.242737711 podStartE2EDuration="1.242737711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:27.241856912 +0000 UTC m=+1264.080199897" watchObservedRunningTime="2026-07-09 18:25:27.242737711 +0000 UTC m=+1264.081080695"
Jul 09 18:25:27.255601 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:27.255575    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"]
Jul 09 18:25:27.257689 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:27.257669    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-04bd4-7dcfc7455-bmt9g"]
Jul 09 18:25:27.932695 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:27.932669    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1" path="/var/lib/kubelet/pods/b8da6f1b-b79a-4e30-be51-e3cd5d6e64f1/volumes"
Jul 09 18:25:33.108662 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:33.108625    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 09 18:25:33.234253 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:33.234227    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:25:36.508231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.508204    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"]
Jul 09 18:25:36.508665 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.508367    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm" podUID="ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" containerName="ensemble-graph-d3ac8" containerID="cri-o://a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51" gracePeriod=30
Jul 09 18:25:36.725665 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.725626    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"]
Jul 09 18:25:36.725904 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.725879    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kserve-container" containerID="cri-o://0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750" gracePeriod=30
Jul 09 18:25:36.725972 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.725944    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kube-rbac-proxy" containerID="cri-o://7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c" gracePeriod=30
Jul 09 18:25:36.727217 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.727196    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"]
Jul 09 18:25:36.732200 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.732180    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:36.735604 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.735582    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fdee6-predictor-serving-cert\""
Jul 09 18:25:36.735694 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.735587    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fdee6-kube-rbac-proxy-sar-config\""
Jul 09 18:25:36.741106 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.741082    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"]
Jul 09 18:25:36.777488 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.777406    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20719484-4189-4cbc-8ee4-f25472086a37-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk\" (UID: \"20719484-4189-4cbc-8ee4-f25472086a37\") " pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:36.777620 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.777495    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20719484-4189-4cbc-8ee4-f25472086a37-proxy-tls\") pod \"success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk\" (UID: \"20719484-4189-4cbc-8ee4-f25472086a37\") " pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:36.777620 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.777587    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntqkd\" (UniqueName: \"kubernetes.io/projected/20719484-4189-4cbc-8ee4-f25472086a37-kube-api-access-ntqkd\") pod \"success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk\" (UID: \"20719484-4189-4cbc-8ee4-f25472086a37\") " pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:36.878397 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.878370    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20719484-4189-4cbc-8ee4-f25472086a37-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk\" (UID: \"20719484-4189-4cbc-8ee4-f25472086a37\") " pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:36.878502 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.878411    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20719484-4189-4cbc-8ee4-f25472086a37-proxy-tls\") pod \"success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk\" (UID: \"20719484-4189-4cbc-8ee4-f25472086a37\") " pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:36.878579 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.878523    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntqkd\" (UniqueName: \"kubernetes.io/projected/20719484-4189-4cbc-8ee4-f25472086a37-kube-api-access-ntqkd\") pod \"success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk\" (UID: \"20719484-4189-4cbc-8ee4-f25472086a37\") " pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:36.879003 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.878983    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20719484-4189-4cbc-8ee4-f25472086a37-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk\" (UID: \"20719484-4189-4cbc-8ee4-f25472086a37\") " pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:36.880763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.880744    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20719484-4189-4cbc-8ee4-f25472086a37-proxy-tls\") pod \"success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk\" (UID: \"20719484-4189-4cbc-8ee4-f25472086a37\") " pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:36.893899 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:36.893876    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntqkd\" (UniqueName: \"kubernetes.io/projected/20719484-4189-4cbc-8ee4-f25472086a37-kube-api-access-ntqkd\") pod \"success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk\" (UID: \"20719484-4189-4cbc-8ee4-f25472086a37\") " pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:37.047257 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:37.047192    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:37.184199 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:37.184172    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"]
Jul 09 18:25:37.186661 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:25:37.186630    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20719484_4189_4cbc_8ee4_f25472086a37.slice/crio-9aaca66ccc0df0d7d8fa2e714dc7079f6f49e56f9d4ab79da809f8a2e3884dfd WatchSource:0}: Error finding container 9aaca66ccc0df0d7d8fa2e714dc7079f6f49e56f9d4ab79da809f8a2e3884dfd: Status 404 returned error can't find the container with id 9aaca66ccc0df0d7d8fa2e714dc7079f6f49e56f9d4ab79da809f8a2e3884dfd
Jul 09 18:25:37.265664 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:37.265640    2572 generic.go:356] "Generic (PLEG): container finished" podID="3beb597f-6558-466c-9da9-4fc0286584b4" containerID="7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c" exitCode=2
Jul 09 18:25:37.265750 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:37.265701    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" event={"ID":"3beb597f-6558-466c-9da9-4fc0286584b4","Type":"ContainerDied","Data":"7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c"}
Jul 09 18:25:37.266997 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:37.266978    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" event={"ID":"20719484-4189-4cbc-8ee4-f25472086a37","Type":"ContainerStarted","Data":"e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655"}
Jul 09 18:25:37.267078 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:37.267022    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" event={"ID":"20719484-4189-4cbc-8ee4-f25472086a37","Type":"ContainerStarted","Data":"9aaca66ccc0df0d7d8fa2e714dc7079f6f49e56f9d4ab79da809f8a2e3884dfd"}
Jul 09 18:25:37.974968 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:37.974932    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 09 18:25:38.233356 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:38.233276    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm" podUID="ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" containerName="ensemble-graph-d3ac8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:25:38.272886 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:38.272855    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" event={"ID":"20719484-4189-4cbc-8ee4-f25472086a37","Type":"ContainerStarted","Data":"db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7"}
Jul 09 18:25:38.273023 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:38.272973    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:38.299547 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:38.299499    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" podStartSLOduration=2.29948618 podStartE2EDuration="2.29948618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:38.296938812 +0000 UTC m=+1275.135281795" watchObservedRunningTime="2026-07-09 18:25:38.29948618 +0000 UTC m=+1275.137829164"
Jul 09 18:25:39.277803 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:39.277776    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:39.278962 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:39.278935    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 18:25:39.574749 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:39.574727    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:25:39.702293 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:39.702264    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9m9w\" (UniqueName: \"kubernetes.io/projected/3beb597f-6558-466c-9da9-4fc0286584b4-kube-api-access-k9m9w\") pod \"3beb597f-6558-466c-9da9-4fc0286584b4\" (UID: \"3beb597f-6558-466c-9da9-4fc0286584b4\") "
Jul 09 18:25:39.702450 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:39.702351    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3beb597f-6558-466c-9da9-4fc0286584b4-kube-rbac-proxy-sar-config\") pod \"3beb597f-6558-466c-9da9-4fc0286584b4\" (UID: \"3beb597f-6558-466c-9da9-4fc0286584b4\") "
Jul 09 18:25:39.702450 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:39.702378    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3beb597f-6558-466c-9da9-4fc0286584b4-proxy-tls\") pod \"3beb597f-6558-466c-9da9-4fc0286584b4\" (UID: \"3beb597f-6558-466c-9da9-4fc0286584b4\") "
Jul 09 18:25:39.702739 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:39.702719    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb597f-6558-466c-9da9-4fc0286584b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3beb597f-6558-466c-9da9-4fc0286584b4" (UID: "3beb597f-6558-466c-9da9-4fc0286584b4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:25:39.704385 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:39.704351    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beb597f-6558-466c-9da9-4fc0286584b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3beb597f-6558-466c-9da9-4fc0286584b4" (UID: "3beb597f-6558-466c-9da9-4fc0286584b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:25:39.704385 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:39.704369    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beb597f-6558-466c-9da9-4fc0286584b4-kube-api-access-k9m9w" (OuterVolumeSpecName: "kube-api-access-k9m9w") pod "3beb597f-6558-466c-9da9-4fc0286584b4" (UID: "3beb597f-6558-466c-9da9-4fc0286584b4"). InnerVolumeSpecName "kube-api-access-k9m9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:25:39.803911 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:39.803854    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3beb597f-6558-466c-9da9-4fc0286584b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:25:39.803911 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:39.803906    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3beb597f-6558-466c-9da9-4fc0286584b4-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:25:39.803911 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:39.803917    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9m9w\" (UniqueName: \"kubernetes.io/projected/3beb597f-6558-466c-9da9-4fc0286584b4-kube-api-access-k9m9w\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:25:40.283265 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.283236    2572 generic.go:356] "Generic (PLEG): container finished" podID="3beb597f-6558-466c-9da9-4fc0286584b4" containerID="0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750" exitCode=0
Jul 09 18:25:40.283656 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.283280    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"
Jul 09 18:25:40.283656 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.283310    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" event={"ID":"3beb597f-6558-466c-9da9-4fc0286584b4","Type":"ContainerDied","Data":"0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750"}
Jul 09 18:25:40.283656 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.283356    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf" event={"ID":"3beb597f-6558-466c-9da9-4fc0286584b4","Type":"ContainerDied","Data":"ba9aa562908eccbc432b13e19b709882ba95bc0ea60f55af096c2bb04449c971"}
Jul 09 18:25:40.283656 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.283383    2572 scope.go:117] "RemoveContainer" containerID="7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c"
Jul 09 18:25:40.283894 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.283828    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 18:25:40.291377 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.291356    2572 scope.go:117] "RemoveContainer" containerID="0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750"
Jul 09 18:25:40.300003 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.299984    2572 scope.go:117] "RemoveContainer" containerID="7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c"
Jul 09 18:25:40.300283 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:25:40.300264    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c\": container with ID starting with 7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c not found: ID does not exist" containerID="7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c"
Jul 09 18:25:40.300378 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.300289    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c"} err="failed to get container status \"7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c\": rpc error: code = NotFound desc = could not find container \"7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c\": container with ID starting with 7c63a3a5115bd328df39802293d75b2b80315644381a56b2697094fdcba12b7c not found: ID does not exist"
Jul 09 18:25:40.300378 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.300303    2572 scope.go:117] "RemoveContainer" containerID="0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750"
Jul 09 18:25:40.300518 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:25:40.300503    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750\": container with ID starting with 0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750 not found: ID does not exist" containerID="0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750"
Jul 09 18:25:40.300596 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.300522    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750"} err="failed to get container status \"0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750\": rpc error: code = NotFound desc = could not find container \"0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750\": container with ID starting with 0b84e88fef06831c1f1e9a0071dc64e7397f0d55dd27bb977782ae8e81cc8750 not found: ID does not exist"
Jul 09 18:25:40.302908 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.302885    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"]
Jul 09 18:25:40.307626 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:40.307607    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3ac8-predictor-5bfd49688b-wskcf"]
Jul 09 18:25:41.932770 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:41.932738    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" path="/var/lib/kubelet/pods/3beb597f-6558-466c-9da9-4fc0286584b4/volumes"
Jul 09 18:25:43.108283 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:43.108255    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:25:43.233396 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:43.233368    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm" podUID="ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" containerName="ensemble-graph-d3ac8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:25:45.287906 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:45.287869    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:25:45.288407 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:45.288383    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 18:25:48.233610 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:48.233572    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm" podUID="ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" containerName="ensemble-graph-d3ac8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:25:48.233984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:48.233683    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:25:53.233287 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:53.233243    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm" podUID="ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" containerName="ensemble-graph-d3ac8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:25:55.289159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:55.289118    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 18:25:56.204005 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.203972    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"]
Jul 09 18:25:56.204318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.204305    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kube-rbac-proxy"
Jul 09 18:25:56.204369 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.204320    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kube-rbac-proxy"
Jul 09 18:25:56.204369 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.204350    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kserve-container"
Jul 09 18:25:56.204369 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.204357    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kserve-container"
Jul 09 18:25:56.204459 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.204410    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kserve-container"
Jul 09 18:25:56.204459 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.204427    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3beb597f-6558-466c-9da9-4fc0286584b4" containerName="kube-rbac-proxy"
Jul 09 18:25:56.206304 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.206288    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:25:56.208732 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.208712    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2e68f-serving-cert\""
Jul 09 18:25:56.208844 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.208822    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2e68f-kube-rbac-proxy-sar-config\""
Jul 09 18:25:56.220527 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.220495    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"]
Jul 09 18:25:56.329345 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.329302    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a220c6f0-7734-4745-b79b-940659a14e09-openshift-service-ca-bundle\") pod \"sequence-graph-2e68f-6bb7f9788b-jxnzx\" (UID: \"a220c6f0-7734-4745-b79b-940659a14e09\") " pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:25:56.329803 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.329365    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a220c6f0-7734-4745-b79b-940659a14e09-proxy-tls\") pod \"sequence-graph-2e68f-6bb7f9788b-jxnzx\" (UID: \"a220c6f0-7734-4745-b79b-940659a14e09\") " pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:25:56.430312 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.430270    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a220c6f0-7734-4745-b79b-940659a14e09-openshift-service-ca-bundle\") pod \"sequence-graph-2e68f-6bb7f9788b-jxnzx\" (UID: \"a220c6f0-7734-4745-b79b-940659a14e09\") " pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:25:56.430502 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.430351    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a220c6f0-7734-4745-b79b-940659a14e09-proxy-tls\") pod \"sequence-graph-2e68f-6bb7f9788b-jxnzx\" (UID: \"a220c6f0-7734-4745-b79b-940659a14e09\") " pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:25:56.430974 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.430950    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a220c6f0-7734-4745-b79b-940659a14e09-openshift-service-ca-bundle\") pod \"sequence-graph-2e68f-6bb7f9788b-jxnzx\" (UID: \"a220c6f0-7734-4745-b79b-940659a14e09\") " pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:25:56.432806 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.432782    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a220c6f0-7734-4745-b79b-940659a14e09-proxy-tls\") pod \"sequence-graph-2e68f-6bb7f9788b-jxnzx\" (UID: \"a220c6f0-7734-4745-b79b-940659a14e09\") " pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:25:56.516015 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.515913    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:25:56.650249 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:56.650220    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"]
Jul 09 18:25:56.652526 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:25:56.652497    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda220c6f0_7734_4745_b79b_940659a14e09.slice/crio-7a4bfca6d3303a1dbd76dbb47ac40ad1aa2b682965ea8d18cf2850d30f88958a WatchSource:0}: Error finding container 7a4bfca6d3303a1dbd76dbb47ac40ad1aa2b682965ea8d18cf2850d30f88958a: Status 404 returned error can't find the container with id 7a4bfca6d3303a1dbd76dbb47ac40ad1aa2b682965ea8d18cf2850d30f88958a
Jul 09 18:25:57.355646 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:57.355604    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx" event={"ID":"a220c6f0-7734-4745-b79b-940659a14e09","Type":"ContainerStarted","Data":"c2bbc5a207a1d8f2635af53c559653cebd69d5d859a10d59e6681ca6642bd094"}
Jul 09 18:25:57.356116 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:57.355663    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx" event={"ID":"a220c6f0-7734-4745-b79b-940659a14e09","Type":"ContainerStarted","Data":"7a4bfca6d3303a1dbd76dbb47ac40ad1aa2b682965ea8d18cf2850d30f88958a"}
Jul 09 18:25:57.356116 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:57.355769    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:25:57.373048 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:57.372976    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx" podStartSLOduration=1.372960004 podStartE2EDuration="1.372960004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:57.371878683 +0000 UTC m=+1294.210221668" watchObservedRunningTime="2026-07-09 18:25:57.372960004 +0000 UTC m=+1294.211303015"
Jul 09 18:25:58.232646 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:25:58.232604    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm" podUID="ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" containerName="ensemble-graph-d3ac8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:26:03.233697 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:03.233649    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm" podUID="ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" containerName="ensemble-graph-d3ac8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:26:03.366075 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:03.366049    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:26:05.288794 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:05.288719    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 18:26:06.304501 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.304462    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"]
Jul 09 18:26:06.304936 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.304732    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx" podUID="a220c6f0-7734-4745-b79b-940659a14e09" containerName="sequence-graph-2e68f" containerID="cri-o://c2bbc5a207a1d8f2635af53c559653cebd69d5d859a10d59e6681ca6642bd094" gracePeriod=30
Jul 09 18:26:06.439002 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.438962    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"]
Jul 09 18:26:06.439473 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.439236    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kserve-container" containerID="cri-o://d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557" gracePeriod=30
Jul 09 18:26:06.439473 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.439312    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kube-rbac-proxy" containerID="cri-o://6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8" gracePeriod=30
Jul 09 18:26:06.512641 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.512608    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"]
Jul 09 18:26:06.516693 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.516662    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:06.519468 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.519438    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4ab35-predictor-serving-cert\""
Jul 09 18:26:06.519629 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.519518    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4ab35-kube-rbac-proxy-sar-config\""
Jul 09 18:26:06.532634 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.527535    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"]
Jul 09 18:26:06.616438 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.616404    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1880d1-02dd-41bc-8c30-81b65371f5b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4ab35-predictor-565c568f4b-ppcbg\" (UID: \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\") " pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:06.616653 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.616497    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgspj\" (UniqueName: \"kubernetes.io/projected/7b1880d1-02dd-41bc-8c30-81b65371f5b4-kube-api-access-zgspj\") pod \"success-200-isvc-4ab35-predictor-565c568f4b-ppcbg\" (UID: \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\") " pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:06.616653 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.616585    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1880d1-02dd-41bc-8c30-81b65371f5b4-proxy-tls\") pod \"success-200-isvc-4ab35-predictor-565c568f4b-ppcbg\" (UID: \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\") " pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:06.706148 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.706121    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:26:06.718015 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.717978    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgspj\" (UniqueName: \"kubernetes.io/projected/7b1880d1-02dd-41bc-8c30-81b65371f5b4-kube-api-access-zgspj\") pod \"success-200-isvc-4ab35-predictor-565c568f4b-ppcbg\" (UID: \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\") " pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:06.718207 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.718023    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1880d1-02dd-41bc-8c30-81b65371f5b4-proxy-tls\") pod \"success-200-isvc-4ab35-predictor-565c568f4b-ppcbg\" (UID: \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\") " pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:06.718207 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.718065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1880d1-02dd-41bc-8c30-81b65371f5b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4ab35-predictor-565c568f4b-ppcbg\" (UID: \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\") " pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:06.718734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.718715    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1880d1-02dd-41bc-8c30-81b65371f5b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4ab35-predictor-565c568f4b-ppcbg\" (UID: \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\") " pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:06.720646 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.720621    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1880d1-02dd-41bc-8c30-81b65371f5b4-proxy-tls\") pod \"success-200-isvc-4ab35-predictor-565c568f4b-ppcbg\" (UID: \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\") " pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:06.726202 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.726165    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgspj\" (UniqueName: \"kubernetes.io/projected/7b1880d1-02dd-41bc-8c30-81b65371f5b4-kube-api-access-zgspj\") pod \"success-200-isvc-4ab35-predictor-565c568f4b-ppcbg\" (UID: \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\") " pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:06.818776 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.818683    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a-proxy-tls\") pod \"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a\" (UID: \"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a\") "
Jul 09 18:26:06.818776 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.818724    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a-openshift-service-ca-bundle\") pod \"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a\" (UID: \"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a\") "
Jul 09 18:26:06.819159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.819129    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" (UID: "ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:06.821031 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.821000    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" (UID: "ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:06.879422 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.879376    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:06.920026 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.919987    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:26:06.920026 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:06.920021    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a-openshift-service-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:26:07.020549 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.020522    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"]
Jul 09 18:26:07.022692 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:26:07.022659    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1880d1_02dd_41bc_8c30_81b65371f5b4.slice/crio-dd5431560c1afe1e61e7d50c3c74252909fb1f3945ba621f7883b820a7cf7859 WatchSource:0}: Error finding container dd5431560c1afe1e61e7d50c3c74252909fb1f3945ba621f7883b820a7cf7859: Status 404 returned error can't find the container with id dd5431560c1afe1e61e7d50c3c74252909fb1f3945ba621f7883b820a7cf7859
Jul 09 18:26:07.398843 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.398750    2572 generic.go:356] "Generic (PLEG): container finished" podID="ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" containerID="a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51" exitCode=0
Jul 09 18:26:07.398843 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.398800    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"
Jul 09 18:26:07.399348 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.398837    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm" event={"ID":"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a","Type":"ContainerDied","Data":"a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51"}
Jul 09 18:26:07.399348 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.398898    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm" event={"ID":"ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a","Type":"ContainerDied","Data":"fc40ce4dae16248dd5125962c5c697670885f0beaff798700d0299d0731c610b"}
Jul 09 18:26:07.399348 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.398923    2572 scope.go:117] "RemoveContainer" containerID="a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51"
Jul 09 18:26:07.400855 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.400831    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" event={"ID":"7b1880d1-02dd-41bc-8c30-81b65371f5b4","Type":"ContainerStarted","Data":"9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f"}
Jul 09 18:26:07.400975 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.400894    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:07.400975 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.400927    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:07.400975 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.400951    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" event={"ID":"7b1880d1-02dd-41bc-8c30-81b65371f5b4","Type":"ContainerStarted","Data":"6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd"}
Jul 09 18:26:07.400975 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.400976    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" event={"ID":"7b1880d1-02dd-41bc-8c30-81b65371f5b4","Type":"ContainerStarted","Data":"dd5431560c1afe1e61e7d50c3c74252909fb1f3945ba621f7883b820a7cf7859"}
Jul 09 18:26:07.402730 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.402705    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:26:07.403176 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.403157    2572 generic.go:356] "Generic (PLEG): container finished" podID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerID="6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8" exitCode=2
Jul 09 18:26:07.403273 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.403198    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" event={"ID":"5f5340ec-c5f9-4511-aa76-c7faf296b184","Type":"ContainerDied","Data":"6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8"}
Jul 09 18:26:07.408736 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.408717    2572 scope.go:117] "RemoveContainer" containerID="a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51"
Jul 09 18:26:07.409035 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:26:07.409016    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51\": container with ID starting with a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51 not found: ID does not exist" containerID="a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51"
Jul 09 18:26:07.409082 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.409044    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51"} err="failed to get container status \"a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51\": rpc error: code = NotFound desc = could not find container \"a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51\": container with ID starting with a56e805dfb7565ec706f3d8a48f0e75cacdf521b50925683074147dc88e5eb51 not found: ID does not exist"
Jul 09 18:26:07.421748 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.421676    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" podStartSLOduration=1.421652635 podStartE2EDuration="1.421652635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:07.420294729 +0000 UTC m=+1304.258637739" watchObservedRunningTime="2026-07-09 18:26:07.421652635 +0000 UTC m=+1304.259995621"
Jul 09 18:26:07.434075 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.434043    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"]
Jul 09 18:26:07.437897 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.437868    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d3ac8-74c8fdd6dd-xmccm"]
Jul 09 18:26:07.933961 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:07.933920    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" path="/var/lib/kubelet/pods/ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a/volumes"
Jul 09 18:26:08.103321 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:08.103278    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 09 18:26:08.364349 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:08.364301    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx" podUID="a220c6f0-7734-4745-b79b-940659a14e09" containerName="sequence-graph-2e68f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:26:08.409367 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:08.409326    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:26:10.888856 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:10.888832    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:26:11.055506 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.055396    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czkxg\" (UniqueName: \"kubernetes.io/projected/5f5340ec-c5f9-4511-aa76-c7faf296b184-kube-api-access-czkxg\") pod \"5f5340ec-c5f9-4511-aa76-c7faf296b184\" (UID: \"5f5340ec-c5f9-4511-aa76-c7faf296b184\") "
Jul 09 18:26:11.055506 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.055455    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f5340ec-c5f9-4511-aa76-c7faf296b184-kube-rbac-proxy-sar-config\") pod \"5f5340ec-c5f9-4511-aa76-c7faf296b184\" (UID: \"5f5340ec-c5f9-4511-aa76-c7faf296b184\") "
Jul 09 18:26:11.055778 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.055593    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f5340ec-c5f9-4511-aa76-c7faf296b184-proxy-tls\") pod \"5f5340ec-c5f9-4511-aa76-c7faf296b184\" (UID: \"5f5340ec-c5f9-4511-aa76-c7faf296b184\") "
Jul 09 18:26:11.055849 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.055824    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5340ec-c5f9-4511-aa76-c7faf296b184-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f5340ec-c5f9-4511-aa76-c7faf296b184" (UID: "5f5340ec-c5f9-4511-aa76-c7faf296b184"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:11.057533 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.057502    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5340ec-c5f9-4511-aa76-c7faf296b184-kube-api-access-czkxg" (OuterVolumeSpecName: "kube-api-access-czkxg") pod "5f5340ec-c5f9-4511-aa76-c7faf296b184" (UID: "5f5340ec-c5f9-4511-aa76-c7faf296b184"). InnerVolumeSpecName "kube-api-access-czkxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:11.057674 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.057605    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5340ec-c5f9-4511-aa76-c7faf296b184-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f5340ec-c5f9-4511-aa76-c7faf296b184" (UID: "5f5340ec-c5f9-4511-aa76-c7faf296b184"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:11.157005 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.156967    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f5340ec-c5f9-4511-aa76-c7faf296b184-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:26:11.157005 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.156998    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czkxg\" (UniqueName: \"kubernetes.io/projected/5f5340ec-c5f9-4511-aa76-c7faf296b184-kube-api-access-czkxg\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:26:11.157005 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.157009    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f5340ec-c5f9-4511-aa76-c7faf296b184-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:26:11.423713 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.423611    2572 generic.go:356] "Generic (PLEG): container finished" podID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerID="d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557" exitCode=0
Jul 09 18:26:11.423713 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.423681    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"
Jul 09 18:26:11.423713 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.423696    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" event={"ID":"5f5340ec-c5f9-4511-aa76-c7faf296b184","Type":"ContainerDied","Data":"d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557"}
Jul 09 18:26:11.423992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.423816    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj" event={"ID":"5f5340ec-c5f9-4511-aa76-c7faf296b184","Type":"ContainerDied","Data":"62d0c3c7d6742e0b8816f95a20c04fa832df0a6760d47f028227747d1f261661"}
Jul 09 18:26:11.423992 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.423858    2572 scope.go:117] "RemoveContainer" containerID="6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8"
Jul 09 18:26:11.432888 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.432869    2572 scope.go:117] "RemoveContainer" containerID="d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557"
Jul 09 18:26:11.440384 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.440359    2572 scope.go:117] "RemoveContainer" containerID="6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8"
Jul 09 18:26:11.440626 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:26:11.440607    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8\": container with ID starting with 6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8 not found: ID does not exist" containerID="6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8"
Jul 09 18:26:11.440703 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.440639    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8"} err="failed to get container status \"6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8\": rpc error: code = NotFound desc = could not find container \"6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8\": container with ID starting with 6b304e1956a7171b47ec2ab2f329d652727cbc8e74a8262743b464a8c6f5b5a8 not found: ID does not exist"
Jul 09 18:26:11.440703 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.440662    2572 scope.go:117] "RemoveContainer" containerID="d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557"
Jul 09 18:26:11.440917 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:26:11.440901    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557\": container with ID starting with d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557 not found: ID does not exist" containerID="d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557"
Jul 09 18:26:11.440962 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.440923    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557"} err="failed to get container status \"d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557\": rpc error: code = NotFound desc = could not find container \"d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557\": container with ID starting with d65342e2b8b804e190ea70ffeddbecc9eb1b069f3e2c9562a9d9e06945ca4557 not found: ID does not exist"
Jul 09 18:26:11.445671 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.445646    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"]
Jul 09 18:26:11.449778 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.449757    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e68f-predictor-96b67d9d6-c2ppj"]
Jul 09 18:26:11.932909 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:11.932879    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" path="/var/lib/kubelet/pods/5f5340ec-c5f9-4511-aa76-c7faf296b184/volumes"
Jul 09 18:26:13.364125 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:13.364086    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx" podUID="a220c6f0-7734-4745-b79b-940659a14e09" containerName="sequence-graph-2e68f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:26:13.414903 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:13.414880    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:26:13.415478 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:13.415452    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:26:15.289067 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:15.289028    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 18:26:18.364859 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:18.364816    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx" podUID="a220c6f0-7734-4745-b79b-940659a14e09" containerName="sequence-graph-2e68f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:26:18.365255 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:18.364933    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:26:23.364312 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:23.364277    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx" podUID="a220c6f0-7734-4745-b79b-940659a14e09" containerName="sequence-graph-2e68f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:26:23.415503 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:23.415474    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:26:25.289721 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:25.289689    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:26:28.369614 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:28.369577    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx" podUID="a220c6f0-7734-4745-b79b-940659a14e09" containerName="sequence-graph-2e68f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:26:33.364055 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:33.364019    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx" podUID="a220c6f0-7734-4745-b79b-940659a14e09" containerName="sequence-graph-2e68f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:26:33.415415 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:33.415375    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:26:36.530343 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.530308    2572 generic.go:356] "Generic (PLEG): container finished" podID="a220c6f0-7734-4745-b79b-940659a14e09" containerID="c2bbc5a207a1d8f2635af53c559653cebd69d5d859a10d59e6681ca6642bd094" exitCode=0
Jul 09 18:26:36.530718 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.530382    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx" event={"ID":"a220c6f0-7734-4745-b79b-940659a14e09","Type":"ContainerDied","Data":"c2bbc5a207a1d8f2635af53c559653cebd69d5d859a10d59e6681ca6642bd094"}
Jul 09 18:26:36.753174 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.753140    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"]
Jul 09 18:26:36.753494 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.753482    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kserve-container"
Jul 09 18:26:36.753538 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.753496    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kserve-container"
Jul 09 18:26:36.753538 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.753507    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" containerName="ensemble-graph-d3ac8"
Jul 09 18:26:36.753538 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.753514    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" containerName="ensemble-graph-d3ac8"
Jul 09 18:26:36.753538 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.753531    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kube-rbac-proxy"
Jul 09 18:26:36.753538 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.753536    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kube-rbac-proxy"
Jul 09 18:26:36.753729 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.753608    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kserve-container"
Jul 09 18:26:36.753729 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.753616    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6d7c47-90d5-4c71-8cdb-80cf301e9b2a" containerName="ensemble-graph-d3ac8"
Jul 09 18:26:36.753729 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.753624    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5340ec-c5f9-4511-aa76-c7faf296b184" containerName="kube-rbac-proxy"
Jul 09 18:26:36.756538 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.756523    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:26:36.759698 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.759681    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-fdee6-serving-cert\""
Jul 09 18:26:36.759784 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.759681    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-fdee6-kube-rbac-proxy-sar-config\""
Jul 09 18:26:36.765077 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.765055    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"]
Jul 09 18:26:36.842983 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.842950    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/213c6204-bea1-45e3-8069-017c3d4514d7-openshift-service-ca-bundle\") pod \"ensemble-graph-fdee6-58647fdc98-8zx69\" (UID: \"213c6204-bea1-45e3-8069-017c3d4514d7\") " pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:26:36.843149 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.843003    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/213c6204-bea1-45e3-8069-017c3d4514d7-proxy-tls\") pod \"ensemble-graph-fdee6-58647fdc98-8zx69\" (UID: \"213c6204-bea1-45e3-8069-017c3d4514d7\") " pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:26:36.943828 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.943801    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/213c6204-bea1-45e3-8069-017c3d4514d7-proxy-tls\") pod \"ensemble-graph-fdee6-58647fdc98-8zx69\" (UID: \"213c6204-bea1-45e3-8069-017c3d4514d7\") " pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:26:36.943932 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.943898    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/213c6204-bea1-45e3-8069-017c3d4514d7-openshift-service-ca-bundle\") pod \"ensemble-graph-fdee6-58647fdc98-8zx69\" (UID: \"213c6204-bea1-45e3-8069-017c3d4514d7\") " pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:26:36.944546 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.944526    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/213c6204-bea1-45e3-8069-017c3d4514d7-openshift-service-ca-bundle\") pod \"ensemble-graph-fdee6-58647fdc98-8zx69\" (UID: \"213c6204-bea1-45e3-8069-017c3d4514d7\") " pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:26:36.946199 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.946176    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/213c6204-bea1-45e3-8069-017c3d4514d7-proxy-tls\") pod \"ensemble-graph-fdee6-58647fdc98-8zx69\" (UID: \"213c6204-bea1-45e3-8069-017c3d4514d7\") " pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:26:36.956619 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:36.956599    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:26:37.044222 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.044191    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a220c6f0-7734-4745-b79b-940659a14e09-openshift-service-ca-bundle\") pod \"a220c6f0-7734-4745-b79b-940659a14e09\" (UID: \"a220c6f0-7734-4745-b79b-940659a14e09\") "
Jul 09 18:26:37.044417 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.044267    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a220c6f0-7734-4745-b79b-940659a14e09-proxy-tls\") pod \"a220c6f0-7734-4745-b79b-940659a14e09\" (UID: \"a220c6f0-7734-4745-b79b-940659a14e09\") "
Jul 09 18:26:37.044586 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.044543    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a220c6f0-7734-4745-b79b-940659a14e09-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a220c6f0-7734-4745-b79b-940659a14e09" (UID: "a220c6f0-7734-4745-b79b-940659a14e09"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:37.046325 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.046304    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a220c6f0-7734-4745-b79b-940659a14e09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a220c6f0-7734-4745-b79b-940659a14e09" (UID: "a220c6f0-7734-4745-b79b-940659a14e09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:37.066841 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.066819    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:26:37.145161 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.145047    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a220c6f0-7734-4745-b79b-940659a14e09-openshift-service-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:26:37.145161 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.145078    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a220c6f0-7734-4745-b79b-940659a14e09-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:26:37.191654 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.191582    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"]
Jul 09 18:26:37.193711 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:26:37.193683    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213c6204_bea1_45e3_8069_017c3d4514d7.slice/crio-c0762b05bb1a407c40f3fc7d36847e506f610c8b55e55a47c0d420c3f4c86f99 WatchSource:0}: Error finding container c0762b05bb1a407c40f3fc7d36847e506f610c8b55e55a47c0d420c3f4c86f99: Status 404 returned error can't find the container with id c0762b05bb1a407c40f3fc7d36847e506f610c8b55e55a47c0d420c3f4c86f99
Jul 09 18:26:37.535628 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.535593    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69" event={"ID":"213c6204-bea1-45e3-8069-017c3d4514d7","Type":"ContainerStarted","Data":"7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c"}
Jul 09 18:26:37.536120 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.535656    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69" event={"ID":"213c6204-bea1-45e3-8069-017c3d4514d7","Type":"ContainerStarted","Data":"c0762b05bb1a407c40f3fc7d36847e506f610c8b55e55a47c0d420c3f4c86f99"}
Jul 09 18:26:37.536120 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.535696    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:26:37.536710 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.536696    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"
Jul 09 18:26:37.536784 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.536707    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx" event={"ID":"a220c6f0-7734-4745-b79b-940659a14e09","Type":"ContainerDied","Data":"7a4bfca6d3303a1dbd76dbb47ac40ad1aa2b682965ea8d18cf2850d30f88958a"}
Jul 09 18:26:37.536784 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.536752    2572 scope.go:117] "RemoveContainer" containerID="c2bbc5a207a1d8f2635af53c559653cebd69d5d859a10d59e6681ca6642bd094"
Jul 09 18:26:37.553335 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.553292    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69" podStartSLOduration=1.553278067 podStartE2EDuration="1.553278067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:37.552760459 +0000 UTC m=+1334.391103446" watchObservedRunningTime="2026-07-09 18:26:37.553278067 +0000 UTC m=+1334.391621050"
Jul 09 18:26:37.567100 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.567071    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"]
Jul 09 18:26:37.571130 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.571109    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2e68f-6bb7f9788b-jxnzx"]
Jul 09 18:26:37.933703 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:37.933625    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a220c6f0-7734-4745-b79b-940659a14e09" path="/var/lib/kubelet/pods/a220c6f0-7734-4745-b79b-940659a14e09/volumes"
Jul 09 18:26:43.415718 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:43.415672    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:26:43.547650 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:43.547621    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:26:53.416292 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:26:53.416216    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:27:06.564470 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.564436    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"]
Jul 09 18:27:06.564948 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.564784    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a220c6f0-7734-4745-b79b-940659a14e09" containerName="sequence-graph-2e68f"
Jul 09 18:27:06.564948 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.564796    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a220c6f0-7734-4745-b79b-940659a14e09" containerName="sequence-graph-2e68f"
Jul 09 18:27:06.564948 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.564864    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a220c6f0-7734-4745-b79b-940659a14e09" containerName="sequence-graph-2e68f"
Jul 09 18:27:06.567763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.567745    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:27:06.570532 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.570509    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-4ab35-kube-rbac-proxy-sar-config\""
Jul 09 18:27:06.570670 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.570567    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-4ab35-serving-cert\""
Jul 09 18:27:06.578189 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.578165    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"]
Jul 09 18:27:06.696708 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.696667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f7c4ab5-3d71-422b-9f7c-44679f6527e5-proxy-tls\") pod \"sequence-graph-4ab35-65f87f5997-7wmgp\" (UID: \"5f7c4ab5-3d71-422b-9f7c-44679f6527e5\") " pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:27:06.696889 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.696769    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7c4ab5-3d71-422b-9f7c-44679f6527e5-openshift-service-ca-bundle\") pod \"sequence-graph-4ab35-65f87f5997-7wmgp\" (UID: \"5f7c4ab5-3d71-422b-9f7c-44679f6527e5\") " pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:27:06.797820 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.797786    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7c4ab5-3d71-422b-9f7c-44679f6527e5-openshift-service-ca-bundle\") pod \"sequence-graph-4ab35-65f87f5997-7wmgp\" (UID: \"5f7c4ab5-3d71-422b-9f7c-44679f6527e5\") " pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:27:06.797995 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.797844    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f7c4ab5-3d71-422b-9f7c-44679f6527e5-proxy-tls\") pod \"sequence-graph-4ab35-65f87f5997-7wmgp\" (UID: \"5f7c4ab5-3d71-422b-9f7c-44679f6527e5\") " pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:27:06.798407 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.798389    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7c4ab5-3d71-422b-9f7c-44679f6527e5-openshift-service-ca-bundle\") pod \"sequence-graph-4ab35-65f87f5997-7wmgp\" (UID: \"5f7c4ab5-3d71-422b-9f7c-44679f6527e5\") " pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:27:06.800182 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.800154    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f7c4ab5-3d71-422b-9f7c-44679f6527e5-proxy-tls\") pod \"sequence-graph-4ab35-65f87f5997-7wmgp\" (UID: \"5f7c4ab5-3d71-422b-9f7c-44679f6527e5\") " pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:27:06.878341 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:06.878253    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:27:07.002576 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:07.002534    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"]
Jul 09 18:27:07.004643 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:27:07.004616    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7c4ab5_3d71_422b_9f7c_44679f6527e5.slice/crio-e5fce2ac9dea7f2ca1283f9f656a423245b0ea6767d6a0c89fc145ee4618f43d WatchSource:0}: Error finding container e5fce2ac9dea7f2ca1283f9f656a423245b0ea6767d6a0c89fc145ee4618f43d: Status 404 returned error can't find the container with id e5fce2ac9dea7f2ca1283f9f656a423245b0ea6767d6a0c89fc145ee4618f43d
Jul 09 18:27:07.658966 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:07.658927    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp" event={"ID":"5f7c4ab5-3d71-422b-9f7c-44679f6527e5","Type":"ContainerStarted","Data":"e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480"}
Jul 09 18:27:07.659373 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:07.658988    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp" event={"ID":"5f7c4ab5-3d71-422b-9f7c-44679f6527e5","Type":"ContainerStarted","Data":"e5fce2ac9dea7f2ca1283f9f656a423245b0ea6767d6a0c89fc145ee4618f43d"}
Jul 09 18:27:07.659373 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:07.659055    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:27:07.680675 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:07.680630    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp" podStartSLOduration=1.6806137429999999 podStartE2EDuration="1.680613743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:27:07.679083581 +0000 UTC m=+1364.517426565" watchObservedRunningTime="2026-07-09 18:27:07.680613743 +0000 UTC m=+1364.518956727"
Jul 09 18:27:13.668295 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:27:13.668265    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:29:23.758594 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:29:23.758528    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:29:23.761058 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:29:23.761034    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:34:23.784615 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:23.784589    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:34:23.789122 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:23.789105    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:34:51.356902 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.356869    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"]
Jul 09 18:34:51.357331 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.357060    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69" podUID="213c6204-bea1-45e3-8069-017c3d4514d7" containerName="ensemble-graph-fdee6" containerID="cri-o://7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c" gracePeriod=30
Jul 09 18:34:51.519725 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.519687    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"]
Jul 09 18:34:51.519982 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.519957    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kserve-container" containerID="cri-o://e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655" gracePeriod=30
Jul 09 18:34:51.520090 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.519972    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kube-rbac-proxy" containerID="cri-o://db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7" gracePeriod=30
Jul 09 18:34:51.671338 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.671312    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"]
Jul 09 18:34:51.675941 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.675917    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:51.678993 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.678972    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-05680-predictor-serving-cert\""
Jul 09 18:34:51.679093 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.679025    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-05680-kube-rbac-proxy-sar-config\""
Jul 09 18:34:51.691383 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.691365    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"]
Jul 09 18:34:51.775305 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.775283    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a53d6908-7679-452b-841c-b2ccb44b7b6b-proxy-tls\") pod \"success-200-isvc-05680-predictor-676fcf4866-kvtsw\" (UID: \"a53d6908-7679-452b-841c-b2ccb44b7b6b\") " pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:51.775406 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.775318    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2k5l\" (UniqueName: \"kubernetes.io/projected/a53d6908-7679-452b-841c-b2ccb44b7b6b-kube-api-access-n2k5l\") pod \"success-200-isvc-05680-predictor-676fcf4866-kvtsw\" (UID: \"a53d6908-7679-452b-841c-b2ccb44b7b6b\") " pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:51.775406 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.775340    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a53d6908-7679-452b-841c-b2ccb44b7b6b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-05680-predictor-676fcf4866-kvtsw\" (UID: \"a53d6908-7679-452b-841c-b2ccb44b7b6b\") " pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:51.876617 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.876593    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2k5l\" (UniqueName: \"kubernetes.io/projected/a53d6908-7679-452b-841c-b2ccb44b7b6b-kube-api-access-n2k5l\") pod \"success-200-isvc-05680-predictor-676fcf4866-kvtsw\" (UID: \"a53d6908-7679-452b-841c-b2ccb44b7b6b\") " pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:51.876737 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.876624    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a53d6908-7679-452b-841c-b2ccb44b7b6b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-05680-predictor-676fcf4866-kvtsw\" (UID: \"a53d6908-7679-452b-841c-b2ccb44b7b6b\") " pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:51.876737 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.876685    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a53d6908-7679-452b-841c-b2ccb44b7b6b-proxy-tls\") pod \"success-200-isvc-05680-predictor-676fcf4866-kvtsw\" (UID: \"a53d6908-7679-452b-841c-b2ccb44b7b6b\") " pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:51.877325 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.877306    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a53d6908-7679-452b-841c-b2ccb44b7b6b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-05680-predictor-676fcf4866-kvtsw\" (UID: \"a53d6908-7679-452b-841c-b2ccb44b7b6b\") " pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:51.878940 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.878923    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a53d6908-7679-452b-841c-b2ccb44b7b6b-proxy-tls\") pod \"success-200-isvc-05680-predictor-676fcf4866-kvtsw\" (UID: \"a53d6908-7679-452b-841c-b2ccb44b7b6b\") " pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:51.886854 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.886799    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2k5l\" (UniqueName: \"kubernetes.io/projected/a53d6908-7679-452b-841c-b2ccb44b7b6b-kube-api-access-n2k5l\") pod \"success-200-isvc-05680-predictor-676fcf4866-kvtsw\" (UID: \"a53d6908-7679-452b-841c-b2ccb44b7b6b\") " pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:51.986067 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:51.986040    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:52.116053 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:52.116031    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"]
Jul 09 18:34:52.118140 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:34:52.118117    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53d6908_7679_452b_841c_b2ccb44b7b6b.slice/crio-f21f1bcd5045f988b1fc2b58f5377ec2a156c90df49a0b4e743958e1b0ad327d WatchSource:0}: Error finding container f21f1bcd5045f988b1fc2b58f5377ec2a156c90df49a0b4e743958e1b0ad327d: Status 404 returned error can't find the container with id f21f1bcd5045f988b1fc2b58f5377ec2a156c90df49a0b4e743958e1b0ad327d
Jul 09 18:34:52.504292 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:52.504260    2572 generic.go:356] "Generic (PLEG): container finished" podID="20719484-4189-4cbc-8ee4-f25472086a37" containerID="db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7" exitCode=2
Jul 09 18:34:52.504764 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:52.504329    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" event={"ID":"20719484-4189-4cbc-8ee4-f25472086a37","Type":"ContainerDied","Data":"db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7"}
Jul 09 18:34:52.506116 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:52.506091    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" event={"ID":"a53d6908-7679-452b-841c-b2ccb44b7b6b","Type":"ContainerStarted","Data":"1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7"}
Jul 09 18:34:52.506247 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:52.506148    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" event={"ID":"a53d6908-7679-452b-841c-b2ccb44b7b6b","Type":"ContainerStarted","Data":"1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518"}
Jul 09 18:34:52.506247 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:52.506177    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" event={"ID":"a53d6908-7679-452b-841c-b2ccb44b7b6b","Type":"ContainerStarted","Data":"f21f1bcd5045f988b1fc2b58f5377ec2a156c90df49a0b4e743958e1b0ad327d"}
Jul 09 18:34:52.506247 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:52.506210    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:52.506247 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:52.506239    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:52.507656 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:52.507630    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 09 18:34:52.529957 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:52.529907    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" podStartSLOduration=1.5298913920000001 podStartE2EDuration="1.529891392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:34:52.52593025 +0000 UTC m=+1829.364273245" watchObservedRunningTime="2026-07-09 18:34:52.529891392 +0000 UTC m=+1829.368234378"
Jul 09 18:34:53.510291 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:53.510249    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 09 18:34:53.546236 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:53.546205    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69" podUID="213c6204-bea1-45e3-8069-017c3d4514d7" containerName="ensemble-graph-fdee6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:34:54.371579 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.371543    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:34:54.496016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.495946    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntqkd\" (UniqueName: \"kubernetes.io/projected/20719484-4189-4cbc-8ee4-f25472086a37-kube-api-access-ntqkd\") pod \"20719484-4189-4cbc-8ee4-f25472086a37\" (UID: \"20719484-4189-4cbc-8ee4-f25472086a37\") "
Jul 09 18:34:54.496016 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.495982    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20719484-4189-4cbc-8ee4-f25472086a37-kube-rbac-proxy-sar-config\") pod \"20719484-4189-4cbc-8ee4-f25472086a37\" (UID: \"20719484-4189-4cbc-8ee4-f25472086a37\") "
Jul 09 18:34:54.496193 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.496033    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20719484-4189-4cbc-8ee4-f25472086a37-proxy-tls\") pod \"20719484-4189-4cbc-8ee4-f25472086a37\" (UID: \"20719484-4189-4cbc-8ee4-f25472086a37\") "
Jul 09 18:34:54.496423 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.496398    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20719484-4189-4cbc-8ee4-f25472086a37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "20719484-4189-4cbc-8ee4-f25472086a37" (UID: "20719484-4189-4cbc-8ee4-f25472086a37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:34:54.498034 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.498016    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20719484-4189-4cbc-8ee4-f25472086a37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "20719484-4189-4cbc-8ee4-f25472086a37" (UID: "20719484-4189-4cbc-8ee4-f25472086a37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:34:54.498082 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.498037    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20719484-4189-4cbc-8ee4-f25472086a37-kube-api-access-ntqkd" (OuterVolumeSpecName: "kube-api-access-ntqkd") pod "20719484-4189-4cbc-8ee4-f25472086a37" (UID: "20719484-4189-4cbc-8ee4-f25472086a37"). InnerVolumeSpecName "kube-api-access-ntqkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:34:54.515080 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.515051    2572 generic.go:356] "Generic (PLEG): container finished" podID="20719484-4189-4cbc-8ee4-f25472086a37" containerID="e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655" exitCode=0
Jul 09 18:34:54.515371 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.515092    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"
Jul 09 18:34:54.515371 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.515120    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" event={"ID":"20719484-4189-4cbc-8ee4-f25472086a37","Type":"ContainerDied","Data":"e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655"}
Jul 09 18:34:54.515371 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.515176    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk" event={"ID":"20719484-4189-4cbc-8ee4-f25472086a37","Type":"ContainerDied","Data":"9aaca66ccc0df0d7d8fa2e714dc7079f6f49e56f9d4ab79da809f8a2e3884dfd"}
Jul 09 18:34:54.515371 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.515204    2572 scope.go:117] "RemoveContainer" containerID="db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7"
Jul 09 18:34:54.523308 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.523290    2572 scope.go:117] "RemoveContainer" containerID="e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655"
Jul 09 18:34:54.530410 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.530395    2572 scope.go:117] "RemoveContainer" containerID="db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7"
Jul 09 18:34:54.530700 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:34:54.530680    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7\": container with ID starting with db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7 not found: ID does not exist" containerID="db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7"
Jul 09 18:34:54.530773 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.530712    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7"} err="failed to get container status \"db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7\": rpc error: code = NotFound desc = could not find container \"db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7\": container with ID starting with db86890d0123ae777627c984d472eb7b28758d425af84b837d487b5a9d1fcbb7 not found: ID does not exist"
Jul 09 18:34:54.530773 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.530735    2572 scope.go:117] "RemoveContainer" containerID="e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655"
Jul 09 18:34:54.530980 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:34:54.530965    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655\": container with ID starting with e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655 not found: ID does not exist" containerID="e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655"
Jul 09 18:34:54.531020 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.530986    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655"} err="failed to get container status \"e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655\": rpc error: code = NotFound desc = could not find container \"e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655\": container with ID starting with e86d208017bb2db349fdecc7dfd3c0f5e6a4704c80630bb23aaa8f55ecde9655 not found: ID does not exist"
Jul 09 18:34:54.540219 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.540198    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"]
Jul 09 18:34:54.547955 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.547936    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fdee6-predictor-9f587cb6d-sjqbk"]
Jul 09 18:34:54.596898 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.596877    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntqkd\" (UniqueName: \"kubernetes.io/projected/20719484-4189-4cbc-8ee4-f25472086a37-kube-api-access-ntqkd\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:34:54.596898 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.596896    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20719484-4189-4cbc-8ee4-f25472086a37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:34:54.597012 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:54.596906    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20719484-4189-4cbc-8ee4-f25472086a37-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:34:55.934597 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:55.934541    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20719484-4189-4cbc-8ee4-f25472086a37" path="/var/lib/kubelet/pods/20719484-4189-4cbc-8ee4-f25472086a37/volumes"
Jul 09 18:34:58.515983 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:58.515946    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:34:58.516396 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:58.516372    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 09 18:34:58.546250 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:34:58.546224    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69" podUID="213c6204-bea1-45e3-8069-017c3d4514d7" containerName="ensemble-graph-fdee6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:35:03.545520 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:03.545485    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69" podUID="213c6204-bea1-45e3-8069-017c3d4514d7" containerName="ensemble-graph-fdee6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:35:03.546118 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:03.545614    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:35:08.516399 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:08.516357    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 09 18:35:08.546363 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:08.546328    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69" podUID="213c6204-bea1-45e3-8069-017c3d4514d7" containerName="ensemble-graph-fdee6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:35:13.545390 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:13.545355    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69" podUID="213c6204-bea1-45e3-8069-017c3d4514d7" containerName="ensemble-graph-fdee6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:35:18.516392 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:18.516350    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 09 18:35:18.545866 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:18.545832    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69" podUID="213c6204-bea1-45e3-8069-017c3d4514d7" containerName="ensemble-graph-fdee6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:35:21.303183 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.303148    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"]
Jul 09 18:35:21.303572 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.303367    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp" podUID="5f7c4ab5-3d71-422b-9f7c-44679f6527e5" containerName="sequence-graph-4ab35" containerID="cri-o://e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480" gracePeriod=30
Jul 09 18:35:21.526104 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.526075    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"]
Jul 09 18:35:21.526391 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.526335    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kserve-container" containerID="cri-o://6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd" gracePeriod=30
Jul 09 18:35:21.526515 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.526383    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kube-rbac-proxy" containerID="cri-o://9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f" gracePeriod=30
Jul 09 18:35:21.534497 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.534477    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:35:21.582439 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.582384    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"]
Jul 09 18:35:21.582746 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.582733    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kserve-container"
Jul 09 18:35:21.582811 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.582748    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kserve-container"
Jul 09 18:35:21.582811 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.582760    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213c6204-bea1-45e3-8069-017c3d4514d7" containerName="ensemble-graph-fdee6"
Jul 09 18:35:21.582811 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.582766    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="213c6204-bea1-45e3-8069-017c3d4514d7" containerName="ensemble-graph-fdee6"
Jul 09 18:35:21.582811 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.582782    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kube-rbac-proxy"
Jul 09 18:35:21.582811 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.582788    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kube-rbac-proxy"
Jul 09 18:35:21.582963 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.582844    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kube-rbac-proxy"
Jul 09 18:35:21.582963 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.582854    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="213c6204-bea1-45e3-8069-017c3d4514d7" containerName="ensemble-graph-fdee6"
Jul 09 18:35:21.582963 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.582861    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="20719484-4189-4cbc-8ee4-f25472086a37" containerName="kserve-container"
Jul 09 18:35:21.585960 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.585946    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:21.591185 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.591168    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-95b80-predictor-serving-cert\""
Jul 09 18:35:21.591493 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.591479    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-95b80-kube-rbac-proxy-sar-config\""
Jul 09 18:35:21.597250 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.597231    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/213c6204-bea1-45e3-8069-017c3d4514d7-openshift-service-ca-bundle\") pod \"213c6204-bea1-45e3-8069-017c3d4514d7\" (UID: \"213c6204-bea1-45e3-8069-017c3d4514d7\") "
Jul 09 18:35:21.597329 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.597283    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/213c6204-bea1-45e3-8069-017c3d4514d7-proxy-tls\") pod \"213c6204-bea1-45e3-8069-017c3d4514d7\" (UID: \"213c6204-bea1-45e3-8069-017c3d4514d7\") "
Jul 09 18:35:21.597635 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.597603    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213c6204-bea1-45e3-8069-017c3d4514d7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "213c6204-bea1-45e3-8069-017c3d4514d7" (UID: "213c6204-bea1-45e3-8069-017c3d4514d7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:35:21.599120 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.599100    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213c6204-bea1-45e3-8069-017c3d4514d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "213c6204-bea1-45e3-8069-017c3d4514d7" (UID: "213c6204-bea1-45e3-8069-017c3d4514d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:35:21.606747 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.606723    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"]
Jul 09 18:35:21.625983 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.625960    2572 generic.go:356] "Generic (PLEG): container finished" podID="213c6204-bea1-45e3-8069-017c3d4514d7" containerID="7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c" exitCode=0
Jul 09 18:35:21.626089 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.626005    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"
Jul 09 18:35:21.626089 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.626044    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69" event={"ID":"213c6204-bea1-45e3-8069-017c3d4514d7","Type":"ContainerDied","Data":"7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c"}
Jul 09 18:35:21.626189 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.626108    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69" event={"ID":"213c6204-bea1-45e3-8069-017c3d4514d7","Type":"ContainerDied","Data":"c0762b05bb1a407c40f3fc7d36847e506f610c8b55e55a47c0d420c3f4c86f99"}
Jul 09 18:35:21.626189 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.626141    2572 scope.go:117] "RemoveContainer" containerID="7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c"
Jul 09 18:35:21.636986 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.636969    2572 scope.go:117] "RemoveContainer" containerID="7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c"
Jul 09 18:35:21.637274 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:35:21.637256    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c\": container with ID starting with 7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c not found: ID does not exist" containerID="7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c"
Jul 09 18:35:21.637323 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.637282    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c"} err="failed to get container status \"7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c\": rpc error: code = NotFound desc = could not find container \"7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c\": container with ID starting with 7738fa566e8cadc8d20cb12a0580f303594cfa8e66ab599270cedf89082e823c not found: ID does not exist"
Jul 09 18:35:21.651318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.651285    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"]
Jul 09 18:35:21.661060 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.661036    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fdee6-58647fdc98-8zx69"]
Jul 09 18:35:21.698263 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.698239    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2qtr\" (UniqueName: \"kubernetes.io/projected/098c38af-c759-44a0-899f-d146526764f5-kube-api-access-g2qtr\") pod \"success-200-isvc-95b80-predictor-87f46f968-cf625\" (UID: \"098c38af-c759-44a0-899f-d146526764f5\") " pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:21.698362 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.698285    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/098c38af-c759-44a0-899f-d146526764f5-proxy-tls\") pod \"success-200-isvc-95b80-predictor-87f46f968-cf625\" (UID: \"098c38af-c759-44a0-899f-d146526764f5\") " pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:21.698413 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.698383    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/098c38af-c759-44a0-899f-d146526764f5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-95b80-predictor-87f46f968-cf625\" (UID: \"098c38af-c759-44a0-899f-d146526764f5\") " pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:21.698462 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.698427    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/213c6204-bea1-45e3-8069-017c3d4514d7-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:35:21.698462 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.698441    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/213c6204-bea1-45e3-8069-017c3d4514d7-openshift-service-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:35:21.799381 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.799353    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2qtr\" (UniqueName: \"kubernetes.io/projected/098c38af-c759-44a0-899f-d146526764f5-kube-api-access-g2qtr\") pod \"success-200-isvc-95b80-predictor-87f46f968-cf625\" (UID: \"098c38af-c759-44a0-899f-d146526764f5\") " pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:21.799492 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.799393    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/098c38af-c759-44a0-899f-d146526764f5-proxy-tls\") pod \"success-200-isvc-95b80-predictor-87f46f968-cf625\" (UID: \"098c38af-c759-44a0-899f-d146526764f5\") " pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:21.799492 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.799451    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/098c38af-c759-44a0-899f-d146526764f5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-95b80-predictor-87f46f968-cf625\" (UID: \"098c38af-c759-44a0-899f-d146526764f5\") " pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:21.800076 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.800054    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/098c38af-c759-44a0-899f-d146526764f5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-95b80-predictor-87f46f968-cf625\" (UID: \"098c38af-c759-44a0-899f-d146526764f5\") " pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:21.801874 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.801854    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/098c38af-c759-44a0-899f-d146526764f5-proxy-tls\") pod \"success-200-isvc-95b80-predictor-87f46f968-cf625\" (UID: \"098c38af-c759-44a0-899f-d146526764f5\") " pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:21.811405 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.811384    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2qtr\" (UniqueName: \"kubernetes.io/projected/098c38af-c759-44a0-899f-d146526764f5-kube-api-access-g2qtr\") pod \"success-200-isvc-95b80-predictor-87f46f968-cf625\" (UID: \"098c38af-c759-44a0-899f-d146526764f5\") " pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:21.895622 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.895541    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:21.935019 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:21.934980    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="213c6204-bea1-45e3-8069-017c3d4514d7" path="/var/lib/kubelet/pods/213c6204-bea1-45e3-8069-017c3d4514d7/volumes"
Jul 09 18:35:22.030355 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:22.030327    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"]
Jul 09 18:35:22.031849 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:35:22.031815    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098c38af_c759_44a0_899f_d146526764f5.slice/crio-7c79975ebe236ca28fc4f6f3a775b1a99fce266896c417d2f7ae0d69398af147 WatchSource:0}: Error finding container 7c79975ebe236ca28fc4f6f3a775b1a99fce266896c417d2f7ae0d69398af147: Status 404 returned error can't find the container with id 7c79975ebe236ca28fc4f6f3a775b1a99fce266896c417d2f7ae0d69398af147
Jul 09 18:35:22.634055 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:22.634016    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" event={"ID":"098c38af-c759-44a0-899f-d146526764f5","Type":"ContainerStarted","Data":"a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6"}
Jul 09 18:35:22.634462 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:22.634083    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" event={"ID":"098c38af-c759-44a0-899f-d146526764f5","Type":"ContainerStarted","Data":"46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5"}
Jul 09 18:35:22.634462 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:22.634111    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" event={"ID":"098c38af-c759-44a0-899f-d146526764f5","Type":"ContainerStarted","Data":"7c79975ebe236ca28fc4f6f3a775b1a99fce266896c417d2f7ae0d69398af147"}
Jul 09 18:35:22.634462 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:22.634142    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:22.635753 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:22.635730    2572 generic.go:356] "Generic (PLEG): container finished" podID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerID="9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f" exitCode=2
Jul 09 18:35:22.635855 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:22.635769    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" event={"ID":"7b1880d1-02dd-41bc-8c30-81b65371f5b4","Type":"ContainerDied","Data":"9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f"}
Jul 09 18:35:22.655996 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:22.655947    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" podStartSLOduration=1.655930648 podStartE2EDuration="1.655930648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:35:22.655679687 +0000 UTC m=+1859.494022665" watchObservedRunningTime="2026-07-09 18:35:22.655930648 +0000 UTC m=+1859.494273634"
Jul 09 18:35:23.410394 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:23.410352    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 09 18:35:23.415854 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:23.415830    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:35:23.639958 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:23.639922    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:23.641304 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:23.641272    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 09 18:35:23.666416 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:23.666355    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp" podUID="5f7c4ab5-3d71-422b-9f7c-44679f6527e5" containerName="sequence-graph-4ab35" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:35:24.277155 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.277129    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:35:24.319794 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.319771    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1880d1-02dd-41bc-8c30-81b65371f5b4-kube-rbac-proxy-sar-config\") pod \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\" (UID: \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\") "
Jul 09 18:35:24.319922 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.319822    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1880d1-02dd-41bc-8c30-81b65371f5b4-proxy-tls\") pod \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\" (UID: \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\") "
Jul 09 18:35:24.319922 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.319846    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgspj\" (UniqueName: \"kubernetes.io/projected/7b1880d1-02dd-41bc-8c30-81b65371f5b4-kube-api-access-zgspj\") pod \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\" (UID: \"7b1880d1-02dd-41bc-8c30-81b65371f5b4\") "
Jul 09 18:35:24.320092 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.320069    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1880d1-02dd-41bc-8c30-81b65371f5b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b1880d1-02dd-41bc-8c30-81b65371f5b4" (UID: "7b1880d1-02dd-41bc-8c30-81b65371f5b4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:35:24.321797 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.321767    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1880d1-02dd-41bc-8c30-81b65371f5b4-kube-api-access-zgspj" (OuterVolumeSpecName: "kube-api-access-zgspj") pod "7b1880d1-02dd-41bc-8c30-81b65371f5b4" (UID: "7b1880d1-02dd-41bc-8c30-81b65371f5b4"). InnerVolumeSpecName "kube-api-access-zgspj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:35:24.321933 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.321914    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1880d1-02dd-41bc-8c30-81b65371f5b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b1880d1-02dd-41bc-8c30-81b65371f5b4" (UID: "7b1880d1-02dd-41bc-8c30-81b65371f5b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:35:24.420714 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.420651    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgspj\" (UniqueName: \"kubernetes.io/projected/7b1880d1-02dd-41bc-8c30-81b65371f5b4-kube-api-access-zgspj\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:35:24.420714 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.420678    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1880d1-02dd-41bc-8c30-81b65371f5b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:35:24.420714 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.420692    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1880d1-02dd-41bc-8c30-81b65371f5b4-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:35:24.645274 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.645243    2572 generic.go:356] "Generic (PLEG): container finished" podID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerID="6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd" exitCode=0
Jul 09 18:35:24.645648 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.645294    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"
Jul 09 18:35:24.645648 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.645326    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" event={"ID":"7b1880d1-02dd-41bc-8c30-81b65371f5b4","Type":"ContainerDied","Data":"6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd"}
Jul 09 18:35:24.645648 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.645401    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg" event={"ID":"7b1880d1-02dd-41bc-8c30-81b65371f5b4","Type":"ContainerDied","Data":"dd5431560c1afe1e61e7d50c3c74252909fb1f3945ba621f7883b820a7cf7859"}
Jul 09 18:35:24.645648 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.645429    2572 scope.go:117] "RemoveContainer" containerID="9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f"
Jul 09 18:35:24.645862 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.645800    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 09 18:35:24.654104 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.654088    2572 scope.go:117] "RemoveContainer" containerID="6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd"
Jul 09 18:35:24.661256 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.661239    2572 scope.go:117] "RemoveContainer" containerID="9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f"
Jul 09 18:35:24.661515 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:35:24.661498    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f\": container with ID starting with 9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f not found: ID does not exist" containerID="9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f"
Jul 09 18:35:24.661599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.661527    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f"} err="failed to get container status \"9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f\": rpc error: code = NotFound desc = could not find container \"9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f\": container with ID starting with 9408ecead5cb34a99e4ff9eb9c37e679cd9f7a38f49244df69b878da1f55eb3f not found: ID does not exist"
Jul 09 18:35:24.661599 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.661568    2572 scope.go:117] "RemoveContainer" containerID="6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd"
Jul 09 18:35:24.661827 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:35:24.661807    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd\": container with ID starting with 6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd not found: ID does not exist" containerID="6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd"
Jul 09 18:35:24.661866 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.661832    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd"} err="failed to get container status \"6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd\": rpc error: code = NotFound desc = could not find container \"6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd\": container with ID starting with 6511ad47223e485340c08dca69eb349bd586ce5157f63a9d09db55fe0cfb60cd not found: ID does not exist"
Jul 09 18:35:24.668802 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.668780    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"]
Jul 09 18:35:24.673059 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:24.673006    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4ab35-predictor-565c568f4b-ppcbg"]
Jul 09 18:35:25.934256 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:25.934226    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" path="/var/lib/kubelet/pods/7b1880d1-02dd-41bc-8c30-81b65371f5b4/volumes"
Jul 09 18:35:28.516794 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:28.516753    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 09 18:35:28.667217 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:28.667187    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp" podUID="5f7c4ab5-3d71-422b-9f7c-44679f6527e5" containerName="sequence-graph-4ab35" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:35:29.650381 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:29.650348    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:35:29.650876 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:29.650839    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 09 18:35:33.666917 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:33.666879    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp" podUID="5f7c4ab5-3d71-422b-9f7c-44679f6527e5" containerName="sequence-graph-4ab35" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:35:33.667255 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:33.667009    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:35:38.516697 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:38.516668    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:35:38.666157 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:38.666123    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp" podUID="5f7c4ab5-3d71-422b-9f7c-44679f6527e5" containerName="sequence-graph-4ab35" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:35:39.651696 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:39.651657    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 09 18:35:43.666725 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:43.666690    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp" podUID="5f7c4ab5-3d71-422b-9f7c-44679f6527e5" containerName="sequence-graph-4ab35" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:35:48.666169 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:48.666086    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp" podUID="5f7c4ab5-3d71-422b-9f7c-44679f6527e5" containerName="sequence-graph-4ab35" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:35:49.651571 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:49.651509    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 09 18:35:51.445374 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.445351    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:35:51.542592 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.542549    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f7c4ab5-3d71-422b-9f7c-44679f6527e5-proxy-tls\") pod \"5f7c4ab5-3d71-422b-9f7c-44679f6527e5\" (UID: \"5f7c4ab5-3d71-422b-9f7c-44679f6527e5\") "
Jul 09 18:35:51.542749 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.542676    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7c4ab5-3d71-422b-9f7c-44679f6527e5-openshift-service-ca-bundle\") pod \"5f7c4ab5-3d71-422b-9f7c-44679f6527e5\" (UID: \"5f7c4ab5-3d71-422b-9f7c-44679f6527e5\") "
Jul 09 18:35:51.543028 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.543003    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7c4ab5-3d71-422b-9f7c-44679f6527e5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5f7c4ab5-3d71-422b-9f7c-44679f6527e5" (UID: "5f7c4ab5-3d71-422b-9f7c-44679f6527e5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:35:51.544563 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.544536    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7c4ab5-3d71-422b-9f7c-44679f6527e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f7c4ab5-3d71-422b-9f7c-44679f6527e5" (UID: "5f7c4ab5-3d71-422b-9f7c-44679f6527e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:35:51.643635 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.643539    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7c4ab5-3d71-422b-9f7c-44679f6527e5-openshift-service-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:35:51.643635 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.643586    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f7c4ab5-3d71-422b-9f7c-44679f6527e5-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:35:51.675453 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.675423    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"]
Jul 09 18:35:51.675757 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.675744    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kserve-container"
Jul 09 18:35:51.675806 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.675759    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kserve-container"
Jul 09 18:35:51.675806 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.675783    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kube-rbac-proxy"
Jul 09 18:35:51.675806 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.675788    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kube-rbac-proxy"
Jul 09 18:35:51.675806 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.675795    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f7c4ab5-3d71-422b-9f7c-44679f6527e5" containerName="sequence-graph-4ab35"
Jul 09 18:35:51.675806 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.675801    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f7c4ab5-3d71-422b-9f7c-44679f6527e5" containerName="sequence-graph-4ab35"
Jul 09 18:35:51.675968 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.675849    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kserve-container"
Jul 09 18:35:51.675968 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.675857    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f7c4ab5-3d71-422b-9f7c-44679f6527e5" containerName="sequence-graph-4ab35"
Jul 09 18:35:51.675968 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.675864    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1880d1-02dd-41bc-8c30-81b65371f5b4" containerName="kube-rbac-proxy"
Jul 09 18:35:51.678638 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.678624    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:35:51.681519 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.681501    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-05680-serving-cert\""
Jul 09 18:35:51.681635 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.681532    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-05680-kube-rbac-proxy-sar-config\""
Jul 09 18:35:51.691382 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.691363    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"]
Jul 09 18:35:51.752091 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.752064    2572 generic.go:356] "Generic (PLEG): container finished" podID="5f7c4ab5-3d71-422b-9f7c-44679f6527e5" containerID="e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480" exitCode=0
Jul 09 18:35:51.752206 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.752106    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"
Jul 09 18:35:51.752206 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.752153    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp" event={"ID":"5f7c4ab5-3d71-422b-9f7c-44679f6527e5","Type":"ContainerDied","Data":"e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480"}
Jul 09 18:35:51.752206 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.752201    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp" event={"ID":"5f7c4ab5-3d71-422b-9f7c-44679f6527e5","Type":"ContainerDied","Data":"e5fce2ac9dea7f2ca1283f9f656a423245b0ea6767d6a0c89fc145ee4618f43d"}
Jul 09 18:35:51.752354 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.752223    2572 scope.go:117] "RemoveContainer" containerID="e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480"
Jul 09 18:35:51.762040 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.762021    2572 scope.go:117] "RemoveContainer" containerID="e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480"
Jul 09 18:35:51.762310 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:35:51.762288    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480\": container with ID starting with e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480 not found: ID does not exist" containerID="e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480"
Jul 09 18:35:51.762394 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.762315    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480"} err="failed to get container status \"e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480\": rpc error: code = NotFound desc = could not find container \"e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480\": container with ID starting with e0d711bd54af1448d28941f1d24d521f6c0d28c4db594bcf5ea163a08567e480 not found: ID does not exist"
Jul 09 18:35:51.782196 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.782169    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"]
Jul 09 18:35:51.785953 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.785935    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4ab35-65f87f5997-7wmgp"]
Jul 09 18:35:51.844929 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.844908    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2e2a578-112b-406c-94f7-e220548d352b-proxy-tls\") pod \"splitter-graph-05680-7c59cd8f45-tj96h\" (UID: \"d2e2a578-112b-406c-94f7-e220548d352b\") " pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:35:51.845019 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.844939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2e2a578-112b-406c-94f7-e220548d352b-openshift-service-ca-bundle\") pod \"splitter-graph-05680-7c59cd8f45-tj96h\" (UID: \"d2e2a578-112b-406c-94f7-e220548d352b\") " pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:35:51.933013 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.932949    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f7c4ab5-3d71-422b-9f7c-44679f6527e5" path="/var/lib/kubelet/pods/5f7c4ab5-3d71-422b-9f7c-44679f6527e5/volumes"
Jul 09 18:35:51.945268 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.945251    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2e2a578-112b-406c-94f7-e220548d352b-proxy-tls\") pod \"splitter-graph-05680-7c59cd8f45-tj96h\" (UID: \"d2e2a578-112b-406c-94f7-e220548d352b\") " pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:35:51.945359 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.945278    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2e2a578-112b-406c-94f7-e220548d352b-openshift-service-ca-bundle\") pod \"splitter-graph-05680-7c59cd8f45-tj96h\" (UID: \"d2e2a578-112b-406c-94f7-e220548d352b\") " pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:35:51.945819 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.945801    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2e2a578-112b-406c-94f7-e220548d352b-openshift-service-ca-bundle\") pod \"splitter-graph-05680-7c59cd8f45-tj96h\" (UID: \"d2e2a578-112b-406c-94f7-e220548d352b\") " pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:35:51.947624 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.947603    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2e2a578-112b-406c-94f7-e220548d352b-proxy-tls\") pod \"splitter-graph-05680-7c59cd8f45-tj96h\" (UID: \"d2e2a578-112b-406c-94f7-e220548d352b\") " pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:35:51.987722 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:51.987703    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:35:52.115119 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:52.115093    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"]
Jul 09 18:35:52.117642 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:35:52.117605    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e2a578_112b_406c_94f7_e220548d352b.slice/crio-b3b3a0d32c2fa5b11903d9acd62037341d93c1a89dc79301476f1a7c958309c1 WatchSource:0}: Error finding container b3b3a0d32c2fa5b11903d9acd62037341d93c1a89dc79301476f1a7c958309c1: Status 404 returned error can't find the container with id b3b3a0d32c2fa5b11903d9acd62037341d93c1a89dc79301476f1a7c958309c1
Jul 09 18:35:52.758410 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:52.758378    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h" event={"ID":"d2e2a578-112b-406c-94f7-e220548d352b","Type":"ContainerStarted","Data":"df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec"}
Jul 09 18:35:52.758801 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:52.758428    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h" event={"ID":"d2e2a578-112b-406c-94f7-e220548d352b","Type":"ContainerStarted","Data":"b3b3a0d32c2fa5b11903d9acd62037341d93c1a89dc79301476f1a7c958309c1"}
Jul 09 18:35:52.758801 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:52.758465    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:35:52.779370 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:52.779320    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h" podStartSLOduration=1.779307299 podStartE2EDuration="1.779307299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:35:52.777604761 +0000 UTC m=+1889.615947749" watchObservedRunningTime="2026-07-09 18:35:52.779307299 +0000 UTC m=+1889.617650282"
Jul 09 18:35:58.768765 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:58.768731    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:35:59.651534 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:35:59.651486    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 09 18:36:01.759590 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:01.759546    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"]
Jul 09 18:36:01.759988 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:01.759736    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h" podUID="d2e2a578-112b-406c-94f7-e220548d352b" containerName="splitter-graph-05680" containerID="cri-o://df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec" gracePeriod=30
Jul 09 18:36:01.946950 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:01.946917    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"]
Jul 09 18:36:01.947162 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:01.947128    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kserve-container" containerID="cri-o://1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518" gracePeriod=30
Jul 09 18:36:01.947229 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:01.947203    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kube-rbac-proxy" containerID="cri-o://1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7" gracePeriod=30
Jul 09 18:36:01.986937 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:01.986912    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"]
Jul 09 18:36:01.990278 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:01.990263    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:01.992803 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:01.992780    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-05318-predictor-serving-cert\""
Jul 09 18:36:01.992899 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:01.992785    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-05318-kube-rbac-proxy-sar-config\""
Jul 09 18:36:02.003744 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.003725    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"]
Jul 09 18:36:02.125895 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.125860    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19951eee-2961-4ffb-a323-1bba72144b1c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") " pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:02.126068 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.125984    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whx2\" (UniqueName: \"kubernetes.io/projected/19951eee-2961-4ffb-a323-1bba72144b1c-kube-api-access-2whx2\") pod \"success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") " pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:02.126145 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.126075    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19951eee-2961-4ffb-a323-1bba72144b1c-proxy-tls\") pod \"success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") " pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:02.226712 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.226680    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whx2\" (UniqueName: \"kubernetes.io/projected/19951eee-2961-4ffb-a323-1bba72144b1c-kube-api-access-2whx2\") pod \"success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") " pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:02.226885 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.226723    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19951eee-2961-4ffb-a323-1bba72144b1c-proxy-tls\") pod \"success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") " pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:02.226885 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.226771    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19951eee-2961-4ffb-a323-1bba72144b1c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") " pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:02.227006 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:36:02.226917    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-05318-predictor-serving-cert: secret "success-200-isvc-05318-predictor-serving-cert" not found
Jul 09 18:36:02.227006 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:36:02.227000    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19951eee-2961-4ffb-a323-1bba72144b1c-proxy-tls podName:19951eee-2961-4ffb-a323-1bba72144b1c nodeName:}" failed. No retries permitted until 2026-07-09 18:36:02.72697801 +0000 UTC m=+1899.565320993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/19951eee-2961-4ffb-a323-1bba72144b1c-proxy-tls") pod "success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" (UID: "19951eee-2961-4ffb-a323-1bba72144b1c") : secret "success-200-isvc-05318-predictor-serving-cert" not found
Jul 09 18:36:02.227441 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.227421    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19951eee-2961-4ffb-a323-1bba72144b1c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") " pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:02.255117 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.255094    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whx2\" (UniqueName: \"kubernetes.io/projected/19951eee-2961-4ffb-a323-1bba72144b1c-kube-api-access-2whx2\") pod \"success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") " pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:02.730874 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.730825    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19951eee-2961-4ffb-a323-1bba72144b1c-proxy-tls\") pod \"success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") " pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:02.733314 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.733297    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19951eee-2961-4ffb-a323-1bba72144b1c-proxy-tls\") pod \"success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") " pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:02.799981 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.799958    2572 generic.go:356] "Generic (PLEG): container finished" podID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerID="1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7" exitCode=2
Jul 09 18:36:02.800270 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.800016    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" event={"ID":"a53d6908-7679-452b-841c-b2ccb44b7b6b","Type":"ContainerDied","Data":"1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7"}
Jul 09 18:36:02.899329 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:02.899289    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:03.044318 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:03.044290    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"]
Jul 09 18:36:03.046770 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:36:03.046740    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19951eee_2961_4ffb_a323_1bba72144b1c.slice/crio-e27c6701c5e63f488f714015e984c1ce54e4da546e454a2691809f0e035c16fb WatchSource:0}: Error finding container e27c6701c5e63f488f714015e984c1ce54e4da546e454a2691809f0e035c16fb: Status 404 returned error can't find the container with id e27c6701c5e63f488f714015e984c1ce54e4da546e454a2691809f0e035c16fb
Jul 09 18:36:03.511120 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:03.511070    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 09 18:36:03.767010 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:03.766920    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h" podUID="d2e2a578-112b-406c-94f7-e220548d352b" containerName="splitter-graph-05680" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:36:03.805723 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:03.805684    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" event={"ID":"19951eee-2961-4ffb-a323-1bba72144b1c","Type":"ContainerStarted","Data":"26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650"}
Jul 09 18:36:03.806211 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:03.805738    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" event={"ID":"19951eee-2961-4ffb-a323-1bba72144b1c","Type":"ContainerStarted","Data":"bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a"}
Jul 09 18:36:03.806211 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:03.805760    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" event={"ID":"19951eee-2961-4ffb-a323-1bba72144b1c","Type":"ContainerStarted","Data":"e27c6701c5e63f488f714015e984c1ce54e4da546e454a2691809f0e035c16fb"}
Jul 09 18:36:03.806211 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:03.805826    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:04.810407 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:04.810376    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:04.811485 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:04.811459    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:36:05.200037 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.200006    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:36:05.228039 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.227975    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" podStartSLOduration=4.227953904 podStartE2EDuration="4.227953904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:36:03.837737817 +0000 UTC m=+1900.676080800" watchObservedRunningTime="2026-07-09 18:36:05.227953904 +0000 UTC m=+1902.066296889"
Jul 09 18:36:05.352535 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.352497    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2k5l\" (UniqueName: \"kubernetes.io/projected/a53d6908-7679-452b-841c-b2ccb44b7b6b-kube-api-access-n2k5l\") pod \"a53d6908-7679-452b-841c-b2ccb44b7b6b\" (UID: \"a53d6908-7679-452b-841c-b2ccb44b7b6b\") "
Jul 09 18:36:05.352744 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.352590    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a53d6908-7679-452b-841c-b2ccb44b7b6b-proxy-tls\") pod \"a53d6908-7679-452b-841c-b2ccb44b7b6b\" (UID: \"a53d6908-7679-452b-841c-b2ccb44b7b6b\") "
Jul 09 18:36:05.352744 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.352655    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a53d6908-7679-452b-841c-b2ccb44b7b6b-kube-rbac-proxy-sar-config\") pod \"a53d6908-7679-452b-841c-b2ccb44b7b6b\" (UID: \"a53d6908-7679-452b-841c-b2ccb44b7b6b\") "
Jul 09 18:36:05.353017 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.352989    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53d6908-7679-452b-841c-b2ccb44b7b6b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a53d6908-7679-452b-841c-b2ccb44b7b6b" (UID: "a53d6908-7679-452b-841c-b2ccb44b7b6b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:36:05.354771 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.354746    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53d6908-7679-452b-841c-b2ccb44b7b6b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a53d6908-7679-452b-841c-b2ccb44b7b6b" (UID: "a53d6908-7679-452b-841c-b2ccb44b7b6b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:36:05.354887 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.354808    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53d6908-7679-452b-841c-b2ccb44b7b6b-kube-api-access-n2k5l" (OuterVolumeSpecName: "kube-api-access-n2k5l") pod "a53d6908-7679-452b-841c-b2ccb44b7b6b" (UID: "a53d6908-7679-452b-841c-b2ccb44b7b6b"). InnerVolumeSpecName "kube-api-access-n2k5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:36:05.453853 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.453824    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2k5l\" (UniqueName: \"kubernetes.io/projected/a53d6908-7679-452b-841c-b2ccb44b7b6b-kube-api-access-n2k5l\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:36:05.453853 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.453857    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a53d6908-7679-452b-841c-b2ccb44b7b6b-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:36:05.453853 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.453868    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a53d6908-7679-452b-841c-b2ccb44b7b6b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:36:05.816008 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.815909    2572 generic.go:356] "Generic (PLEG): container finished" podID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerID="1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518" exitCode=0
Jul 09 18:36:05.816008 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.815963    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"
Jul 09 18:36:05.816527 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.816000    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" event={"ID":"a53d6908-7679-452b-841c-b2ccb44b7b6b","Type":"ContainerDied","Data":"1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518"}
Jul 09 18:36:05.816527 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.816069    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw" event={"ID":"a53d6908-7679-452b-841c-b2ccb44b7b6b","Type":"ContainerDied","Data":"f21f1bcd5045f988b1fc2b58f5377ec2a156c90df49a0b4e743958e1b0ad327d"}
Jul 09 18:36:05.816527 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.816112    2572 scope.go:117] "RemoveContainer" containerID="1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7"
Jul 09 18:36:05.816702 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.816631    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:36:05.825941 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.825914    2572 scope.go:117] "RemoveContainer" containerID="1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518"
Jul 09 18:36:05.839871 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.839846    2572 scope.go:117] "RemoveContainer" containerID="1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7"
Jul 09 18:36:05.840223 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:36:05.840202    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7\": container with ID starting with 1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7 not found: ID does not exist" containerID="1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7"
Jul 09 18:36:05.840265 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.840234    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7"} err="failed to get container status \"1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7\": rpc error: code = NotFound desc = could not find container \"1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7\": container with ID starting with 1bd9f91b43f81bd640cdafc84a5ffcef4f44e5a5b392978d20e435872d78d6a7 not found: ID does not exist"
Jul 09 18:36:05.840265 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.840255    2572 scope.go:117] "RemoveContainer" containerID="1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518"
Jul 09 18:36:05.840529 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:36:05.840509    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518\": container with ID starting with 1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518 not found: ID does not exist" containerID="1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518"
Jul 09 18:36:05.840608 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.840534    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518"} err="failed to get container status \"1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518\": rpc error: code = NotFound desc = could not find container \"1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518\": container with ID starting with 1d586886fc06d678c10598ae360a457ba1145bb745dee36d5c897ea1b239e518 not found: ID does not exist"
Jul 09 18:36:05.845407 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.845375    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"]
Jul 09 18:36:05.849452 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.849418    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-05680-predictor-676fcf4866-kvtsw"]
Jul 09 18:36:05.934287 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:05.934243    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" path="/var/lib/kubelet/pods/a53d6908-7679-452b-841c-b2ccb44b7b6b/volumes"
Jul 09 18:36:08.766247 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:08.766209    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h" podUID="d2e2a578-112b-406c-94f7-e220548d352b" containerName="splitter-graph-05680" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:36:09.651506 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:09.651475    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:36:10.821513 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:10.821483    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:36:10.822082 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:10.822055    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:36:13.766359 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:13.766320    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h" podUID="d2e2a578-112b-406c-94f7-e220548d352b" containerName="splitter-graph-05680" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:36:13.766749 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:13.766420    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:36:18.766715 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:18.766673    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h" podUID="d2e2a578-112b-406c-94f7-e220548d352b" containerName="splitter-graph-05680" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:36:20.822924 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:20.822878    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:36:21.593117 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.593079    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"]
Jul 09 18:36:21.593526 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.593495    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kube-rbac-proxy"
Jul 09 18:36:21.593643 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.593527    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kube-rbac-proxy"
Jul 09 18:36:21.593643 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.593572    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kserve-container"
Jul 09 18:36:21.593643 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.593582    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kserve-container"
Jul 09 18:36:21.593793 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.593668    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kserve-container"
Jul 09 18:36:21.593793 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.593684    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53d6908-7679-452b-841c-b2ccb44b7b6b" containerName="kube-rbac-proxy"
Jul 09 18:36:21.598217 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.598196    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:36:21.601488 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.601468    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-95b80-kube-rbac-proxy-sar-config\""
Jul 09 18:36:21.601621 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.601599    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-95b80-serving-cert\""
Jul 09 18:36:21.608795 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.608773    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"]
Jul 09 18:36:21.697643 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.697611    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa9566e6-f514-457b-8b6a-7d8c1d757790-openshift-service-ca-bundle\") pod \"switch-graph-95b80-6db858f4b6-rs95t\" (UID: \"fa9566e6-f514-457b-8b6a-7d8c1d757790\") " pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:36:21.697643 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.697645    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9566e6-f514-457b-8b6a-7d8c1d757790-proxy-tls\") pod \"switch-graph-95b80-6db858f4b6-rs95t\" (UID: \"fa9566e6-f514-457b-8b6a-7d8c1d757790\") " pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:36:21.798119 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.798088    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa9566e6-f514-457b-8b6a-7d8c1d757790-openshift-service-ca-bundle\") pod \"switch-graph-95b80-6db858f4b6-rs95t\" (UID: \"fa9566e6-f514-457b-8b6a-7d8c1d757790\") " pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:36:21.798119 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.798120    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9566e6-f514-457b-8b6a-7d8c1d757790-proxy-tls\") pod \"switch-graph-95b80-6db858f4b6-rs95t\" (UID: \"fa9566e6-f514-457b-8b6a-7d8c1d757790\") " pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:36:21.798310 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:36:21.798216    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-95b80-serving-cert: secret "switch-graph-95b80-serving-cert" not found
Jul 09 18:36:21.798310 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:36:21.798275    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa9566e6-f514-457b-8b6a-7d8c1d757790-proxy-tls podName:fa9566e6-f514-457b-8b6a-7d8c1d757790 nodeName:}" failed. No retries permitted until 2026-07-09 18:36:22.298259116 +0000 UTC m=+1919.136602078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fa9566e6-f514-457b-8b6a-7d8c1d757790-proxy-tls") pod "switch-graph-95b80-6db858f4b6-rs95t" (UID: "fa9566e6-f514-457b-8b6a-7d8c1d757790") : secret "switch-graph-95b80-serving-cert" not found
Jul 09 18:36:21.798724 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:21.798707    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa9566e6-f514-457b-8b6a-7d8c1d757790-openshift-service-ca-bundle\") pod \"switch-graph-95b80-6db858f4b6-rs95t\" (UID: \"fa9566e6-f514-457b-8b6a-7d8c1d757790\") " pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:36:22.303129 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:22.303084    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9566e6-f514-457b-8b6a-7d8c1d757790-proxy-tls\") pod \"switch-graph-95b80-6db858f4b6-rs95t\" (UID: \"fa9566e6-f514-457b-8b6a-7d8c1d757790\") " pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:36:22.305334 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:22.305315    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9566e6-f514-457b-8b6a-7d8c1d757790-proxy-tls\") pod \"switch-graph-95b80-6db858f4b6-rs95t\" (UID: \"fa9566e6-f514-457b-8b6a-7d8c1d757790\") " pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:36:22.509500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:22.509471    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:36:22.645573 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:22.645519    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"]
Jul 09 18:36:22.649724 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:36:22.649696    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9566e6_f514_457b_8b6a_7d8c1d757790.slice/crio-bf8f745df47c7fa862facd324caae38c3a14f29d02875c2c47af80f35eb72a2e WatchSource:0}: Error finding container bf8f745df47c7fa862facd324caae38c3a14f29d02875c2c47af80f35eb72a2e: Status 404 returned error can't find the container with id bf8f745df47c7fa862facd324caae38c3a14f29d02875c2c47af80f35eb72a2e
Jul 09 18:36:22.898087 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:22.897990    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t" event={"ID":"fa9566e6-f514-457b-8b6a-7d8c1d757790","Type":"ContainerStarted","Data":"bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946"}
Jul 09 18:36:22.898087 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:22.898050    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:36:22.898087 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:22.898071    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t" event={"ID":"fa9566e6-f514-457b-8b6a-7d8c1d757790","Type":"ContainerStarted","Data":"bf8f745df47c7fa862facd324caae38c3a14f29d02875c2c47af80f35eb72a2e"}
Jul 09 18:36:23.766151 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:23.766113    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h" podUID="d2e2a578-112b-406c-94f7-e220548d352b" containerName="splitter-graph-05680" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:36:28.766578 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:28.766531    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h" podUID="d2e2a578-112b-406c-94f7-e220548d352b" containerName="splitter-graph-05680" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:36:28.907805 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:28.907782    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:36:28.930785 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:28.930735    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t" podStartSLOduration=7.930719998 podStartE2EDuration="7.930719998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:36:22.92258511 +0000 UTC m=+1919.760928109" watchObservedRunningTime="2026-07-09 18:36:28.930719998 +0000 UTC m=+1925.769062982"
Jul 09 18:36:30.822404 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:30.822363    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:36:31.901747 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:31.901722    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:36:31.937579 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:31.937541    2572 generic.go:356] "Generic (PLEG): container finished" podID="d2e2a578-112b-406c-94f7-e220548d352b" containerID="df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec" exitCode=0
Jul 09 18:36:31.937691 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:31.937594    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"
Jul 09 18:36:31.937691 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:31.937596    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h" event={"ID":"d2e2a578-112b-406c-94f7-e220548d352b","Type":"ContainerDied","Data":"df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec"}
Jul 09 18:36:31.937691 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:31.937651    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h" event={"ID":"d2e2a578-112b-406c-94f7-e220548d352b","Type":"ContainerDied","Data":"b3b3a0d32c2fa5b11903d9acd62037341d93c1a89dc79301476f1a7c958309c1"}
Jul 09 18:36:31.937691 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:31.937684    2572 scope.go:117] "RemoveContainer" containerID="df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec"
Jul 09 18:36:31.948078 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:31.948051    2572 scope.go:117] "RemoveContainer" containerID="df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec"
Jul 09 18:36:31.948354 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:36:31.948337    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec\": container with ID starting with df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec not found: ID does not exist" containerID="df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec"
Jul 09 18:36:31.948409 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:31.948362    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec"} err="failed to get container status \"df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec\": rpc error: code = NotFound desc = could not find container \"df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec\": container with ID starting with df49be6f2737e34f15a72ed227fcd677f244aca8a146c278b87781a08c1918ec not found: ID does not exist"
Jul 09 18:36:31.979674 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:31.979651    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2e2a578-112b-406c-94f7-e220548d352b-proxy-tls\") pod \"d2e2a578-112b-406c-94f7-e220548d352b\" (UID: \"d2e2a578-112b-406c-94f7-e220548d352b\") "
Jul 09 18:36:31.979763 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:31.979690    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2e2a578-112b-406c-94f7-e220548d352b-openshift-service-ca-bundle\") pod \"d2e2a578-112b-406c-94f7-e220548d352b\" (UID: \"d2e2a578-112b-406c-94f7-e220548d352b\") "
Jul 09 18:36:31.980051 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:31.980032    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e2a578-112b-406c-94f7-e220548d352b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d2e2a578-112b-406c-94f7-e220548d352b" (UID: "d2e2a578-112b-406c-94f7-e220548d352b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:36:31.981546 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:31.981507    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e2a578-112b-406c-94f7-e220548d352b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2e2a578-112b-406c-94f7-e220548d352b" (UID: "d2e2a578-112b-406c-94f7-e220548d352b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:36:32.080748 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:32.080691    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2e2a578-112b-406c-94f7-e220548d352b-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:36:32.080748 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:32.080715    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2e2a578-112b-406c-94f7-e220548d352b-openshift-service-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:36:32.260672 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:32.260647    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"]
Jul 09 18:36:32.265378 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:32.265358    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-05680-7c59cd8f45-tj96h"]
Jul 09 18:36:33.932630 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:33.932597    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e2a578-112b-406c-94f7-e220548d352b" path="/var/lib/kubelet/pods/d2e2a578-112b-406c-94f7-e220548d352b/volumes"
Jul 09 18:36:40.823072 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:40.823025    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:36:50.822804 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:36:50.822769    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:37:02.077582 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.077537    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"]
Jul 09 18:37:02.078002 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.077940    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e2a578-112b-406c-94f7-e220548d352b" containerName="splitter-graph-05680"
Jul 09 18:37:02.078002 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.077953    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e2a578-112b-406c-94f7-e220548d352b" containerName="splitter-graph-05680"
Jul 09 18:37:02.078002 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.078003    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e2a578-112b-406c-94f7-e220548d352b" containerName="splitter-graph-05680"
Jul 09 18:37:02.080681 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.080660    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:37:02.084636 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.084616    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-05318-serving-cert\""
Jul 09 18:37:02.084738 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.084616    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-05318-kube-rbac-proxy-sar-config\""
Jul 09 18:37:02.101839 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.101809    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"]
Jul 09 18:37:02.127482 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.127456    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-openshift-service-ca-bundle\") pod \"splitter-graph-05318-8558c6757d-frzqc\" (UID: \"aa5633d3-6eb4-4694-9bde-c01fd5e5710d\") " pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:37:02.127622 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.127498    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-proxy-tls\") pod \"splitter-graph-05318-8558c6757d-frzqc\" (UID: \"aa5633d3-6eb4-4694-9bde-c01fd5e5710d\") " pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:37:02.228703 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.228670    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-proxy-tls\") pod \"splitter-graph-05318-8558c6757d-frzqc\" (UID: \"aa5633d3-6eb4-4694-9bde-c01fd5e5710d\") " pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:37:02.228860 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.228780    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-openshift-service-ca-bundle\") pod \"splitter-graph-05318-8558c6757d-frzqc\" (UID: \"aa5633d3-6eb4-4694-9bde-c01fd5e5710d\") " pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:37:02.228860 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:37:02.228826    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-05318-serving-cert: secret "splitter-graph-05318-serving-cert" not found
Jul 09 18:37:02.228942 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:37:02.228926    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-proxy-tls podName:aa5633d3-6eb4-4694-9bde-c01fd5e5710d nodeName:}" failed. No retries permitted until 2026-07-09 18:37:02.728904209 +0000 UTC m=+1959.567247174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-proxy-tls") pod "splitter-graph-05318-8558c6757d-frzqc" (UID: "aa5633d3-6eb4-4694-9bde-c01fd5e5710d") : secret "splitter-graph-05318-serving-cert" not found
Jul 09 18:37:02.229421 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.229403    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-openshift-service-ca-bundle\") pod \"splitter-graph-05318-8558c6757d-frzqc\" (UID: \"aa5633d3-6eb4-4694-9bde-c01fd5e5710d\") " pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:37:02.731649 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.731612    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-proxy-tls\") pod \"splitter-graph-05318-8558c6757d-frzqc\" (UID: \"aa5633d3-6eb4-4694-9bde-c01fd5e5710d\") " pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:37:02.733996 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.733975    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-proxy-tls\") pod \"splitter-graph-05318-8558c6757d-frzqc\" (UID: \"aa5633d3-6eb4-4694-9bde-c01fd5e5710d\") " pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:37:02.990721 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:02.990638    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:37:03.119163 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:03.119134    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"]
Jul 09 18:37:03.122305 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:37:03.122279    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5633d3_6eb4_4694_9bde_c01fd5e5710d.slice/crio-94ab40a602bcf61e71df0cf34357f8bf6af58b92f642005e6a5b22b2c97f2fb9 WatchSource:0}: Error finding container 94ab40a602bcf61e71df0cf34357f8bf6af58b92f642005e6a5b22b2c97f2fb9: Status 404 returned error can't find the container with id 94ab40a602bcf61e71df0cf34357f8bf6af58b92f642005e6a5b22b2c97f2fb9
Jul 09 18:37:04.075082 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:04.075052    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc" event={"ID":"aa5633d3-6eb4-4694-9bde-c01fd5e5710d","Type":"ContainerStarted","Data":"64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8"}
Jul 09 18:37:04.075245 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:04.075098    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc" event={"ID":"aa5633d3-6eb4-4694-9bde-c01fd5e5710d","Type":"ContainerStarted","Data":"94ab40a602bcf61e71df0cf34357f8bf6af58b92f642005e6a5b22b2c97f2fb9"}
Jul 09 18:37:04.075245 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:04.075183    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:37:04.094000 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:04.093960    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc" podStartSLOduration=2.093947703 podStartE2EDuration="2.093947703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:04.092501404 +0000 UTC m=+1960.930844389" watchObservedRunningTime="2026-07-09 18:37:04.093947703 +0000 UTC m=+1960.932290691"
Jul 09 18:37:10.084903 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:37:10.084865    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:39:23.809939 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:39:23.809909    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:39:23.815264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:39:23.815243    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:44:23.836596 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:44:23.836478    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:44:23.847965 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:44:23.847945    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:45:16.689660 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:16.689549    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"]
Jul 09 18:45:16.690156 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:16.689811    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc" podUID="aa5633d3-6eb4-4694-9bde-c01fd5e5710d" containerName="splitter-graph-05318" containerID="cri-o://64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8" gracePeriod=30
Jul 09 18:45:16.831383 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:16.831358    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"]
Jul 09 18:45:16.831595 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:16.831567    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kserve-container" containerID="cri-o://bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a" gracePeriod=30
Jul 09 18:45:16.831727 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:16.831587    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kube-rbac-proxy" containerID="cri-o://26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650" gracePeriod=30
Jul 09 18:45:17.044409 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:17.044377    2572 generic.go:356] "Generic (PLEG): container finished" podID="19951eee-2961-4ffb-a323-1bba72144b1c" containerID="26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650" exitCode=2
Jul 09 18:45:17.044521 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:17.044424    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" event={"ID":"19951eee-2961-4ffb-a323-1bba72144b1c","Type":"ContainerDied","Data":"26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650"}
Jul 09 18:45:19.481413 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:19.481391    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:45:19.549150 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:19.549102    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19951eee-2961-4ffb-a323-1bba72144b1c-kube-rbac-proxy-sar-config\") pod \"19951eee-2961-4ffb-a323-1bba72144b1c\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") "
Jul 09 18:45:19.549150 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:19.549165    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2whx2\" (UniqueName: \"kubernetes.io/projected/19951eee-2961-4ffb-a323-1bba72144b1c-kube-api-access-2whx2\") pod \"19951eee-2961-4ffb-a323-1bba72144b1c\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") "
Jul 09 18:45:19.549420 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:19.549231    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19951eee-2961-4ffb-a323-1bba72144b1c-proxy-tls\") pod \"19951eee-2961-4ffb-a323-1bba72144b1c\" (UID: \"19951eee-2961-4ffb-a323-1bba72144b1c\") "
Jul 09 18:45:19.549521 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:19.549496    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19951eee-2961-4ffb-a323-1bba72144b1c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19951eee-2961-4ffb-a323-1bba72144b1c" (UID: "19951eee-2961-4ffb-a323-1bba72144b1c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:45:19.551339 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:19.551316    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19951eee-2961-4ffb-a323-1bba72144b1c-kube-api-access-2whx2" (OuterVolumeSpecName: "kube-api-access-2whx2") pod "19951eee-2961-4ffb-a323-1bba72144b1c" (UID: "19951eee-2961-4ffb-a323-1bba72144b1c"). InnerVolumeSpecName "kube-api-access-2whx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:45:19.551416 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:19.551334    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19951eee-2961-4ffb-a323-1bba72144b1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19951eee-2961-4ffb-a323-1bba72144b1c" (UID: "19951eee-2961-4ffb-a323-1bba72144b1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:45:19.650484 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:19.650394    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2whx2\" (UniqueName: \"kubernetes.io/projected/19951eee-2961-4ffb-a323-1bba72144b1c-kube-api-access-2whx2\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:45:19.650484 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:19.650433    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19951eee-2961-4ffb-a323-1bba72144b1c-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:45:19.650484 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:19.650443    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19951eee-2961-4ffb-a323-1bba72144b1c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:45:20.058163 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.058136    2572 generic.go:356] "Generic (PLEG): container finished" podID="19951eee-2961-4ffb-a323-1bba72144b1c" containerID="bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a" exitCode=0
Jul 09 18:45:20.058324 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.058192    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"
Jul 09 18:45:20.058324 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.058217    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" event={"ID":"19951eee-2961-4ffb-a323-1bba72144b1c","Type":"ContainerDied","Data":"bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a"}
Jul 09 18:45:20.058324 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.058278    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9" event={"ID":"19951eee-2961-4ffb-a323-1bba72144b1c","Type":"ContainerDied","Data":"e27c6701c5e63f488f714015e984c1ce54e4da546e454a2691809f0e035c16fb"}
Jul 09 18:45:20.058324 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.058305    2572 scope.go:117] "RemoveContainer" containerID="26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650"
Jul 09 18:45:20.066101 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.066082    2572 scope.go:117] "RemoveContainer" containerID="bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a"
Jul 09 18:45:20.073175 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.073160    2572 scope.go:117] "RemoveContainer" containerID="26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650"
Jul 09 18:45:20.073403 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:45:20.073388    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650\": container with ID starting with 26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650 not found: ID does not exist" containerID="26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650"
Jul 09 18:45:20.073451 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.073411    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650"} err="failed to get container status \"26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650\": rpc error: code = NotFound desc = could not find container \"26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650\": container with ID starting with 26c31e28595abfbc718b287386af876f377c0fb0d89a81f09c29ea141a3c3650 not found: ID does not exist"
Jul 09 18:45:20.073451 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.073426    2572 scope.go:117] "RemoveContainer" containerID="bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a"
Jul 09 18:45:20.073655 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:45:20.073636    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a\": container with ID starting with bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a not found: ID does not exist" containerID="bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a"
Jul 09 18:45:20.073703 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.073659    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a"} err="failed to get container status \"bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a\": rpc error: code = NotFound desc = could not find container \"bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a\": container with ID starting with bf6512881d1ae4ae6c22ef1c0d8757264ec316021bcab29b1b6808047954a26a not found: ID does not exist"
Jul 09 18:45:20.078422 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.078403    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"]
Jul 09 18:45:20.083321 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.083301    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc" podUID="aa5633d3-6eb4-4694-9bde-c01fd5e5710d" containerName="splitter-graph-05318" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:45:20.086353 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:20.086333    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-05318-predictor-6c8d5ffd46-8pjm9"]
Jul 09 18:45:21.933905 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:21.933867    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" path="/var/lib/kubelet/pods/19951eee-2961-4ffb-a323-1bba72144b1c/volumes"
Jul 09 18:45:25.083606 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:25.083528    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc" podUID="aa5633d3-6eb4-4694-9bde-c01fd5e5710d" containerName="splitter-graph-05318" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:45:30.083845 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:30.083807    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc" podUID="aa5633d3-6eb4-4694-9bde-c01fd5e5710d" containerName="splitter-graph-05318" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:45:30.084220 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:30.083934    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:45:35.084031 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:35.083994    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc" podUID="aa5633d3-6eb4-4694-9bde-c01fd5e5710d" containerName="splitter-graph-05318" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:45:40.083261 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:40.083222    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc" podUID="aa5633d3-6eb4-4694-9bde-c01fd5e5710d" containerName="splitter-graph-05318" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:45:45.084057 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:45.084022    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc" podUID="aa5633d3-6eb4-4694-9bde-c01fd5e5710d" containerName="splitter-graph-05318" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:45:46.831826 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:46.831800    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:45:46.960792 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:46.960767    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-openshift-service-ca-bundle\") pod \"aa5633d3-6eb4-4694-9bde-c01fd5e5710d\" (UID: \"aa5633d3-6eb4-4694-9bde-c01fd5e5710d\") "
Jul 09 18:45:46.960958 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:46.960837    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-proxy-tls\") pod \"aa5633d3-6eb4-4694-9bde-c01fd5e5710d\" (UID: \"aa5633d3-6eb4-4694-9bde-c01fd5e5710d\") "
Jul 09 18:45:46.961170 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:46.961144    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "aa5633d3-6eb4-4694-9bde-c01fd5e5710d" (UID: "aa5633d3-6eb4-4694-9bde-c01fd5e5710d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:45:46.963005 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:46.962982    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa5633d3-6eb4-4694-9bde-c01fd5e5710d" (UID: "aa5633d3-6eb4-4694-9bde-c01fd5e5710d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:45:47.061516 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:47.061489    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:45:47.061516 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:47.061514    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5633d3-6eb4-4694-9bde-c01fd5e5710d-openshift-service-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:45:47.171884 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:47.171857    2572 generic.go:356] "Generic (PLEG): container finished" podID="aa5633d3-6eb4-4694-9bde-c01fd5e5710d" containerID="64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8" exitCode=0
Jul 09 18:45:47.171996 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:47.171908    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"
Jul 09 18:45:47.171996 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:47.171946    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc" event={"ID":"aa5633d3-6eb4-4694-9bde-c01fd5e5710d","Type":"ContainerDied","Data":"64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8"}
Jul 09 18:45:47.171996 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:47.171995    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc" event={"ID":"aa5633d3-6eb4-4694-9bde-c01fd5e5710d","Type":"ContainerDied","Data":"94ab40a602bcf61e71df0cf34357f8bf6af58b92f642005e6a5b22b2c97f2fb9"}
Jul 09 18:45:47.172132 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:47.172016    2572 scope.go:117] "RemoveContainer" containerID="64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8"
Jul 09 18:45:47.180325 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:47.180309    2572 scope.go:117] "RemoveContainer" containerID="64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8"
Jul 09 18:45:47.180588 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:45:47.180543    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8\": container with ID starting with 64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8 not found: ID does not exist" containerID="64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8"
Jul 09 18:45:47.180655 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:47.180600    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8"} err="failed to get container status \"64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8\": rpc error: code = NotFound desc = could not find container \"64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8\": container with ID starting with 64d8e4053c40c60114243a8a8dd9f56e17333d22b9b0b6bf5dc37b13b9ec5ce8 not found: ID does not exist"
Jul 09 18:45:47.224042 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:47.223985    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"]
Jul 09 18:45:47.227395 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:47.227376    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-05318-8558c6757d-frzqc"]
Jul 09 18:45:47.933133 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:45:47.933105    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa5633d3-6eb4-4694-9bde-c01fd5e5710d" path="/var/lib/kubelet/pods/aa5633d3-6eb4-4694-9bde-c01fd5e5710d/volumes"
Jul 09 18:49:23.867781 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:49:23.867676    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:49:23.872734 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:49:23.872713    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:52:40.567375 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:40.567305    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"]
Jul 09 18:52:40.567876 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:40.567490    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t" podUID="fa9566e6-f514-457b-8b6a-7d8c1d757790" containerName="switch-graph-95b80" containerID="cri-o://bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946" gracePeriod=30
Jul 09 18:52:40.721544 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:40.721513    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"]
Jul 09 18:52:40.721794 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:40.721769    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kserve-container" containerID="cri-o://46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5" gracePeriod=30
Jul 09 18:52:40.721875 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:40.721798    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kube-rbac-proxy" containerID="cri-o://a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6" gracePeriod=30
Jul 09 18:52:41.790094 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:41.790058    2572 generic.go:356] "Generic (PLEG): container finished" podID="098c38af-c759-44a0-899f-d146526764f5" containerID="a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6" exitCode=2
Jul 09 18:52:41.790453 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:41.790116    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" event={"ID":"098c38af-c759-44a0-899f-d146526764f5","Type":"ContainerDied","Data":"a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6"}
Jul 09 18:52:42.145096 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.145017    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5jbsj/must-gather-4zlm2"]
Jul 09 18:52:42.145343 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.145329    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa5633d3-6eb4-4694-9bde-c01fd5e5710d" containerName="splitter-graph-05318"
Jul 09 18:52:42.145387 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.145345    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa5633d3-6eb4-4694-9bde-c01fd5e5710d" containerName="splitter-graph-05318"
Jul 09 18:52:42.145387 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.145367    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kserve-container"
Jul 09 18:52:42.145387 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.145372    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kserve-container"
Jul 09 18:52:42.145387 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.145381    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kube-rbac-proxy"
Jul 09 18:52:42.145387 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.145386    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kube-rbac-proxy"
Jul 09 18:52:42.145542 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.145432    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kube-rbac-proxy"
Jul 09 18:52:42.145542 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.145441    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa5633d3-6eb4-4694-9bde-c01fd5e5710d" containerName="splitter-graph-05318"
Jul 09 18:52:42.145542 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.145448    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="19951eee-2961-4ffb-a323-1bba72144b1c" containerName="kserve-container"
Jul 09 18:52:42.148359 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.148343    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jbsj/must-gather-4zlm2"
Jul 09 18:52:42.151121 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.151100    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5jbsj\"/\"default-dockercfg-9bdft\""
Jul 09 18:52:42.151121 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.151111    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5jbsj\"/\"openshift-service-ca.crt\""
Jul 09 18:52:42.151282 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.151103    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5jbsj\"/\"kube-root-ca.crt\""
Jul 09 18:52:42.158789 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.158766    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jbsj/must-gather-4zlm2"]
Jul 09 18:52:42.215769 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.215740    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/87188b36-06f0-41e4-8cb7-bc676c308f7a-must-gather-output\") pod \"must-gather-4zlm2\" (UID: \"87188b36-06f0-41e4-8cb7-bc676c308f7a\") " pod="openshift-must-gather-5jbsj/must-gather-4zlm2"
Jul 09 18:52:42.215929 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.215791    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pkmz\" (UniqueName: \"kubernetes.io/projected/87188b36-06f0-41e4-8cb7-bc676c308f7a-kube-api-access-8pkmz\") pod \"must-gather-4zlm2\" (UID: \"87188b36-06f0-41e4-8cb7-bc676c308f7a\") " pod="openshift-must-gather-5jbsj/must-gather-4zlm2"
Jul 09 18:52:42.316634 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.316609    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/87188b36-06f0-41e4-8cb7-bc676c308f7a-must-gather-output\") pod \"must-gather-4zlm2\" (UID: \"87188b36-06f0-41e4-8cb7-bc676c308f7a\") " pod="openshift-must-gather-5jbsj/must-gather-4zlm2"
Jul 09 18:52:42.316752 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.316652    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pkmz\" (UniqueName: \"kubernetes.io/projected/87188b36-06f0-41e4-8cb7-bc676c308f7a-kube-api-access-8pkmz\") pod \"must-gather-4zlm2\" (UID: \"87188b36-06f0-41e4-8cb7-bc676c308f7a\") " pod="openshift-must-gather-5jbsj/must-gather-4zlm2"
Jul 09 18:52:42.316935 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.316917    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/87188b36-06f0-41e4-8cb7-bc676c308f7a-must-gather-output\") pod \"must-gather-4zlm2\" (UID: \"87188b36-06f0-41e4-8cb7-bc676c308f7a\") " pod="openshift-must-gather-5jbsj/must-gather-4zlm2"
Jul 09 18:52:42.324489 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.324457    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pkmz\" (UniqueName: \"kubernetes.io/projected/87188b36-06f0-41e4-8cb7-bc676c308f7a-kube-api-access-8pkmz\") pod \"must-gather-4zlm2\" (UID: \"87188b36-06f0-41e4-8cb7-bc676c308f7a\") " pod="openshift-must-gather-5jbsj/must-gather-4zlm2"
Jul 09 18:52:42.469428 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.469404    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jbsj/must-gather-4zlm2"
Jul 09 18:52:42.597184 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.597159    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jbsj/must-gather-4zlm2"]
Jul 09 18:52:42.599440 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:52:42.599412    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87188b36_06f0_41e4_8cb7_bc676c308f7a.slice/crio-48f8091c641e65cca7be720dc85a94971de15cdaf0704b775adab812946bd040 WatchSource:0}: Error finding container 48f8091c641e65cca7be720dc85a94971de15cdaf0704b775adab812946bd040: Status 404 returned error can't find the container with id 48f8091c641e65cca7be720dc85a94971de15cdaf0704b775adab812946bd040
Jul 09 18:52:42.601347 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.601331    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:52:42.795975 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:42.795902    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jbsj/must-gather-4zlm2" event={"ID":"87188b36-06f0-41e4-8cb7-bc676c308f7a","Type":"ContainerStarted","Data":"48f8091c641e65cca7be720dc85a94971de15cdaf0704b775adab812946bd040"}
Jul 09 18:52:43.250501 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.250481    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:52:43.324843 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.324815    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2qtr\" (UniqueName: \"kubernetes.io/projected/098c38af-c759-44a0-899f-d146526764f5-kube-api-access-g2qtr\") pod \"098c38af-c759-44a0-899f-d146526764f5\" (UID: \"098c38af-c759-44a0-899f-d146526764f5\") "
Jul 09 18:52:43.324843 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.324850    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/098c38af-c759-44a0-899f-d146526764f5-proxy-tls\") pod \"098c38af-c759-44a0-899f-d146526764f5\" (UID: \"098c38af-c759-44a0-899f-d146526764f5\") "
Jul 09 18:52:43.325091 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.324875    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/098c38af-c759-44a0-899f-d146526764f5-kube-rbac-proxy-sar-config\") pod \"098c38af-c759-44a0-899f-d146526764f5\" (UID: \"098c38af-c759-44a0-899f-d146526764f5\") "
Jul 09 18:52:43.325303 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.325276    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098c38af-c759-44a0-899f-d146526764f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "098c38af-c759-44a0-899f-d146526764f5" (UID: "098c38af-c759-44a0-899f-d146526764f5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:52:43.326934 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.326910    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098c38af-c759-44a0-899f-d146526764f5-kube-api-access-g2qtr" (OuterVolumeSpecName: "kube-api-access-g2qtr") pod "098c38af-c759-44a0-899f-d146526764f5" (UID: "098c38af-c759-44a0-899f-d146526764f5"). InnerVolumeSpecName "kube-api-access-g2qtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:52:43.327011 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.326911    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098c38af-c759-44a0-899f-d146526764f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "098c38af-c759-44a0-899f-d146526764f5" (UID: "098c38af-c759-44a0-899f-d146526764f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:52:43.425810 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.425694    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2qtr\" (UniqueName: \"kubernetes.io/projected/098c38af-c759-44a0-899f-d146526764f5-kube-api-access-g2qtr\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:52:43.425810 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.425727    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/098c38af-c759-44a0-899f-d146526764f5-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:52:43.425810 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.425742    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/098c38af-c759-44a0-899f-d146526764f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:52:43.804114 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.804076    2572 generic.go:356] "Generic (PLEG): container finished" podID="098c38af-c759-44a0-899f-d146526764f5" containerID="46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5" exitCode=0
Jul 09 18:52:43.804590 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.804150    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" event={"ID":"098c38af-c759-44a0-899f-d146526764f5","Type":"ContainerDied","Data":"46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5"}
Jul 09 18:52:43.804590 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.804209    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625" event={"ID":"098c38af-c759-44a0-899f-d146526764f5","Type":"ContainerDied","Data":"7c79975ebe236ca28fc4f6f3a775b1a99fce266896c417d2f7ae0d69398af147"}
Jul 09 18:52:43.804590 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.804249    2572 scope.go:117] "RemoveContainer" containerID="a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6"
Jul 09 18:52:43.804590 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.804369    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"
Jul 09 18:52:43.818101 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.816827    2572 scope.go:117] "RemoveContainer" containerID="46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5"
Jul 09 18:52:43.828873 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.828835    2572 scope.go:117] "RemoveContainer" containerID="a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6"
Jul 09 18:52:43.829398 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:52:43.829364    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6\": container with ID starting with a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6 not found: ID does not exist" containerID="a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6"
Jul 09 18:52:43.829510 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.829407    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6"} err="failed to get container status \"a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6\": rpc error: code = NotFound desc = could not find container \"a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6\": container with ID starting with a9e93072d354d7d6d222aa39fd81f4f65714d995b2d9535dc4f9cf6b028735d6 not found: ID does not exist"
Jul 09 18:52:43.829510 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.829430    2572 scope.go:117] "RemoveContainer" containerID="46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5"
Jul 09 18:52:43.830810 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:52:43.830695    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5\": container with ID starting with 46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5 not found: ID does not exist" containerID="46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5"
Jul 09 18:52:43.830810 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.830728    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5"} err="failed to get container status \"46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5\": rpc error: code = NotFound desc = could not find container \"46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5\": container with ID starting with 46bccf4c4bde534a049bc259879edaf04ff61d427410fee7f295cf4119289fc5 not found: ID does not exist"
Jul 09 18:52:43.835947 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.835904    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"]
Jul 09 18:52:43.840619 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.840595    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-95b80-predictor-87f46f968-cf625"]
Jul 09 18:52:43.907791 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.907749    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t" podUID="fa9566e6-f514-457b-8b6a-7d8c1d757790" containerName="switch-graph-95b80" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:52:43.934878 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:43.934843    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="098c38af-c759-44a0-899f-d146526764f5" path="/var/lib/kubelet/pods/098c38af-c759-44a0-899f-d146526764f5/volumes"
Jul 09 18:52:48.906977 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:48.906934    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t" podUID="fa9566e6-f514-457b-8b6a-7d8c1d757790" containerName="switch-graph-95b80" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:52:49.833930 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:49.833894    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jbsj/must-gather-4zlm2" event={"ID":"87188b36-06f0-41e4-8cb7-bc676c308f7a","Type":"ContainerStarted","Data":"ab79c689f194c1e8d1827344ac42251fb351cfd87304b5fb20d7d23b2004ad9e"}
Jul 09 18:52:49.834103 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:49.833945    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jbsj/must-gather-4zlm2" event={"ID":"87188b36-06f0-41e4-8cb7-bc676c308f7a","Type":"ContainerStarted","Data":"d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0"}
Jul 09 18:52:49.853873 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:49.853825    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5jbsj/must-gather-4zlm2" podStartSLOduration=1.417567162 podStartE2EDuration="7.853810774s" totalImagesPullingTime="6.436243612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:52:49.85193697 +0000 UTC m=+2906.690279966" watchObservedRunningTime="2026-07-09 18:52:49.853810774 +0000 UTC m=+2906.692153757"
Jul 09 18:52:53.905803 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:53.905764    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t" podUID="fa9566e6-f514-457b-8b6a-7d8c1d757790" containerName="switch-graph-95b80" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:52:53.906240 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:53.905888    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:52:57.701971 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:57.701934    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:52:58.554032 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:58.553994    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:52:58.905862 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:58.905758    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t" podUID="fa9566e6-f514-457b-8b6a-7d8c1d757790" containerName="switch-graph-95b80" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:52:59.403344 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:52:59.403309    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:53:00.228025 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:00.227988    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:53:01.048781 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:01.048752    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:53:01.869968 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:01.869934    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:53:02.670668 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:02.670644    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:53:03.459479 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:03.459455    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:53:03.906023 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:03.905914    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t" podUID="fa9566e6-f514-457b-8b6a-7d8c1d757790" containerName="switch-graph-95b80" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:53:04.298523 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:04.298488    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:53:05.124622 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:05.124592    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:53:05.893896 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:05.893859    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:53:06.720159 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:06.720126    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:53:07.529523 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:07.529485    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:53:08.381073 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:08.381043    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-95b80-6db858f4b6-rs95t_fa9566e6-f514-457b-8b6a-7d8c1d757790/switch-graph-95b80/0.log"
Jul 09 18:53:08.907796 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:08.907760    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t" podUID="fa9566e6-f514-457b-8b6a-7d8c1d757790" containerName="switch-graph-95b80" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:53:10.739167 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.739143    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:53:10.855759 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.855674    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa9566e6-f514-457b-8b6a-7d8c1d757790-openshift-service-ca-bundle\") pod \"fa9566e6-f514-457b-8b6a-7d8c1d757790\" (UID: \"fa9566e6-f514-457b-8b6a-7d8c1d757790\") "
Jul 09 18:53:10.855914 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.855799    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9566e6-f514-457b-8b6a-7d8c1d757790-proxy-tls\") pod \"fa9566e6-f514-457b-8b6a-7d8c1d757790\" (UID: \"fa9566e6-f514-457b-8b6a-7d8c1d757790\") "
Jul 09 18:53:10.856076 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.856054    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9566e6-f514-457b-8b6a-7d8c1d757790-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fa9566e6-f514-457b-8b6a-7d8c1d757790" (UID: "fa9566e6-f514-457b-8b6a-7d8c1d757790"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:53:10.857764 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.857734    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa9566e6-f514-457b-8b6a-7d8c1d757790-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa9566e6-f514-457b-8b6a-7d8c1d757790" (UID: "fa9566e6-f514-457b-8b6a-7d8c1d757790"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:53:10.924591 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.924542    2572 generic.go:356] "Generic (PLEG): container finished" podID="fa9566e6-f514-457b-8b6a-7d8c1d757790" containerID="bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946" exitCode=0
Jul 09 18:53:10.924747 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.924585    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t" event={"ID":"fa9566e6-f514-457b-8b6a-7d8c1d757790","Type":"ContainerDied","Data":"bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946"}
Jul 09 18:53:10.924747 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.924604    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"
Jul 09 18:53:10.924747 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.924649    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t" event={"ID":"fa9566e6-f514-457b-8b6a-7d8c1d757790","Type":"ContainerDied","Data":"bf8f745df47c7fa862facd324caae38c3a14f29d02875c2c47af80f35eb72a2e"}
Jul 09 18:53:10.924747 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.924681    2572 scope.go:117] "RemoveContainer" containerID="bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946"
Jul 09 18:53:10.929649 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.929505    2572 generic.go:356] "Generic (PLEG): container finished" podID="87188b36-06f0-41e4-8cb7-bc676c308f7a" containerID="d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0" exitCode=0
Jul 09 18:53:10.929785 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.929653    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jbsj/must-gather-4zlm2" event={"ID":"87188b36-06f0-41e4-8cb7-bc676c308f7a","Type":"ContainerDied","Data":"d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0"}
Jul 09 18:53:10.929980 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.929963    2572 scope.go:117] "RemoveContainer" containerID="d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0"
Jul 09 18:53:10.934878 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.934863    2572 scope.go:117] "RemoveContainer" containerID="bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946"
Jul 09 18:53:10.935124 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:53:10.935106    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946\": container with ID starting with bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946 not found: ID does not exist" containerID="bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946"
Jul 09 18:53:10.935175 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.935132    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946"} err="failed to get container status \"bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946\": rpc error: code = NotFound desc = could not find container \"bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946\": container with ID starting with bf29c91cd74ed5844eb1fe8a5086a1743704c55d23a0388cffbe5273bf717946 not found: ID does not exist"
Jul 09 18:53:10.956702 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.956682    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9566e6-f514-457b-8b6a-7d8c1d757790-proxy-tls\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:53:10.956702 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.956702    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa9566e6-f514-457b-8b6a-7d8c1d757790-openshift-service-ca-bundle\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:53:10.961117 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.961096    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"]
Jul 09 18:53:10.964717 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:10.964698    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-95b80-6db858f4b6-rs95t"]
Jul 09 18:53:11.596316 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:11.596291    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5jbsj_must-gather-4zlm2_87188b36-06f0-41e4-8cb7-bc676c308f7a/gather/0.log"
Jul 09 18:53:11.934389 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:11.934312    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa9566e6-f514-457b-8b6a-7d8c1d757790" path="/var/lib/kubelet/pods/fa9566e6-f514-457b-8b6a-7d8c1d757790/volumes"
Jul 09 18:53:12.190141 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.190074    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gkb77/must-gather-hl9gf"]
Jul 09 18:53:12.190378 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.190366    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa9566e6-f514-457b-8b6a-7d8c1d757790" containerName="switch-graph-95b80"
Jul 09 18:53:12.190424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.190378    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa9566e6-f514-457b-8b6a-7d8c1d757790" containerName="switch-graph-95b80"
Jul 09 18:53:12.190424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.190391    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kserve-container"
Jul 09 18:53:12.190424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.190396    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kserve-container"
Jul 09 18:53:12.190424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.190411    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kube-rbac-proxy"
Jul 09 18:53:12.190424 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.190416    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kube-rbac-proxy"
Jul 09 18:53:12.190624 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.190471    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kube-rbac-proxy"
Jul 09 18:53:12.190624 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.190480    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="098c38af-c759-44a0-899f-d146526764f5" containerName="kserve-container"
Jul 09 18:53:12.190624 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.190488    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa9566e6-f514-457b-8b6a-7d8c1d757790" containerName="switch-graph-95b80"
Jul 09 18:53:12.193288 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.193266    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gkb77/must-gather-hl9gf"
Jul 09 18:53:12.196989 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.196969    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gkb77\"/\"default-dockercfg-9z2x5\""
Jul 09 18:53:12.197100 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.196970    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gkb77\"/\"openshift-service-ca.crt\""
Jul 09 18:53:12.197222 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.197207    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gkb77\"/\"kube-root-ca.crt\""
Jul 09 18:53:12.205487 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.205466    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gkb77/must-gather-hl9gf"]
Jul 09 18:53:12.366258 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.366231    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/241c7723-612f-470c-805b-87ce37903210-must-gather-output\") pod \"must-gather-hl9gf\" (UID: \"241c7723-612f-470c-805b-87ce37903210\") " pod="openshift-must-gather-gkb77/must-gather-hl9gf"
Jul 09 18:53:12.366391 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.366271    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjwdg\" (UniqueName: \"kubernetes.io/projected/241c7723-612f-470c-805b-87ce37903210-kube-api-access-cjwdg\") pod \"must-gather-hl9gf\" (UID: \"241c7723-612f-470c-805b-87ce37903210\") " pod="openshift-must-gather-gkb77/must-gather-hl9gf"
Jul 09 18:53:12.467065 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.467042    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/241c7723-612f-470c-805b-87ce37903210-must-gather-output\") pod \"must-gather-hl9gf\" (UID: \"241c7723-612f-470c-805b-87ce37903210\") " pod="openshift-must-gather-gkb77/must-gather-hl9gf"
Jul 09 18:53:12.467187 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.467082    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjwdg\" (UniqueName: \"kubernetes.io/projected/241c7723-612f-470c-805b-87ce37903210-kube-api-access-cjwdg\") pod \"must-gather-hl9gf\" (UID: \"241c7723-612f-470c-805b-87ce37903210\") " pod="openshift-must-gather-gkb77/must-gather-hl9gf"
Jul 09 18:53:12.467367 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.467350    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/241c7723-612f-470c-805b-87ce37903210-must-gather-output\") pod \"must-gather-hl9gf\" (UID: \"241c7723-612f-470c-805b-87ce37903210\") " pod="openshift-must-gather-gkb77/must-gather-hl9gf"
Jul 09 18:53:12.476076 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.476052    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjwdg\" (UniqueName: \"kubernetes.io/projected/241c7723-612f-470c-805b-87ce37903210-kube-api-access-cjwdg\") pod \"must-gather-hl9gf\" (UID: \"241c7723-612f-470c-805b-87ce37903210\") " pod="openshift-must-gather-gkb77/must-gather-hl9gf"
Jul 09 18:53:12.501938 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.501899    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gkb77/must-gather-hl9gf"
Jul 09 18:53:12.629747 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.629723    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gkb77/must-gather-hl9gf"]
Jul 09 18:53:12.631845 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:53:12.631807    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod241c7723_612f_470c_805b_87ce37903210.slice/crio-6d432c5a5e301061ad13a45635da6683a97984aea0f2494a895040c7e4803f39 WatchSource:0}: Error finding container 6d432c5a5e301061ad13a45635da6683a97984aea0f2494a895040c7e4803f39: Status 404 returned error can't find the container with id 6d432c5a5e301061ad13a45635da6683a97984aea0f2494a895040c7e4803f39
Jul 09 18:53:12.944278 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:12.944192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gkb77/must-gather-hl9gf" event={"ID":"241c7723-612f-470c-805b-87ce37903210","Type":"ContainerStarted","Data":"6d432c5a5e301061ad13a45635da6683a97984aea0f2494a895040c7e4803f39"}
Jul 09 18:53:13.951610 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:13.951575    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gkb77/must-gather-hl9gf" event={"ID":"241c7723-612f-470c-805b-87ce37903210","Type":"ContainerStarted","Data":"914c8bc5fbefa500bef6964f35592b52eba9671c00813053f6a77a054b01f8a2"}
Jul 09 18:53:13.951610 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:13.951614    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gkb77/must-gather-hl9gf" event={"ID":"241c7723-612f-470c-805b-87ce37903210","Type":"ContainerStarted","Data":"b035f142897279ee369167266dc14593c0dbc82bfbb965efdd0c5f4b33280308"}
Jul 09 18:53:13.970081 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:13.970022    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gkb77/must-gather-hl9gf" podStartSLOduration=1.22446722 podStartE2EDuration="1.970000269s" totalImagesPullingTime="745.533049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:13.969146032 +0000 UTC m=+2930.807489018" watchObservedRunningTime="2026-07-09 18:53:13.970000269 +0000 UTC m=+2930.808343245"
Jul 09 18:53:14.856401 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:14.856371    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j6pks_f0f42946-27c2-47be-b09b-cefcd2f8ce8e/global-pull-secret-syncer/0.log"
Jul 09 18:53:14.997349 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:14.997318    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-srlzs_64637b8e-a4e3-44fc-bd3d-c69743d5bcfa/konnectivity-agent/0.log"
Jul 09 18:53:15.053517 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:15.053490    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-107.ec2.internal_c54f981040f96dfb2de508fe0606b9e4/haproxy/0.log"
Jul 09 18:53:17.032994 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.032960    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-5jbsj/must-gather-4zlm2"]
Jul 09 18:53:17.033994 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.033964    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-5jbsj/must-gather-4zlm2" podUID="87188b36-06f0-41e4-8cb7-bc676c308f7a" containerName="copy" containerID="cri-o://ab79c689f194c1e8d1827344ac42251fb351cfd87304b5fb20d7d23b2004ad9e" gracePeriod=2
Jul 09 18:53:17.048926 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.048727    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-5jbsj/must-gather-4zlm2"]
Jul 09 18:53:17.391022 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.390358    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5jbsj_must-gather-4zlm2_87188b36-06f0-41e4-8cb7-bc676c308f7a/copy/0.log"
Jul 09 18:53:17.392920 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.392659    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jbsj/must-gather-4zlm2"
Jul 09 18:53:17.399569 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:53:17.398677    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4zlm2\" is forbidden: User \"system:node:ip-10-0-130-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5jbsj\": no relationship found between node 'ip-10-0-130-107.ec2.internal' and this object" podUID="87188b36-06f0-41e4-8cb7-bc676c308f7a" pod="openshift-must-gather-5jbsj/must-gather-4zlm2"
Jul 09 18:53:17.418641 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.415634    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/87188b36-06f0-41e4-8cb7-bc676c308f7a-must-gather-output\") pod \"87188b36-06f0-41e4-8cb7-bc676c308f7a\" (UID: \"87188b36-06f0-41e4-8cb7-bc676c308f7a\") "
Jul 09 18:53:17.418641 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.415731    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pkmz\" (UniqueName: \"kubernetes.io/projected/87188b36-06f0-41e4-8cb7-bc676c308f7a-kube-api-access-8pkmz\") pod \"87188b36-06f0-41e4-8cb7-bc676c308f7a\" (UID: \"87188b36-06f0-41e4-8cb7-bc676c308f7a\") "
Jul 09 18:53:17.418918 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.418880    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87188b36-06f0-41e4-8cb7-bc676c308f7a-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "87188b36-06f0-41e4-8cb7-bc676c308f7a" (UID: "87188b36-06f0-41e4-8cb7-bc676c308f7a"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:53:17.422571 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.419110    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87188b36-06f0-41e4-8cb7-bc676c308f7a-kube-api-access-8pkmz" (OuterVolumeSpecName: "kube-api-access-8pkmz") pod "87188b36-06f0-41e4-8cb7-bc676c308f7a" (UID: "87188b36-06f0-41e4-8cb7-bc676c308f7a"). InnerVolumeSpecName "kube-api-access-8pkmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:53:17.516709 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.516669    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pkmz\" (UniqueName: \"kubernetes.io/projected/87188b36-06f0-41e4-8cb7-bc676c308f7a-kube-api-access-8pkmz\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:53:17.516709 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.516709    2572 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/87188b36-06f0-41e4-8cb7-bc676c308f7a-must-gather-output\") on node \"ip-10-0-130-107.ec2.internal\" DevicePath \"\""
Jul 09 18:53:17.950986 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.950945    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87188b36-06f0-41e4-8cb7-bc676c308f7a" path="/var/lib/kubelet/pods/87188b36-06f0-41e4-8cb7-bc676c308f7a/volumes"
Jul 09 18:53:17.973984 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.973905    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5jbsj_must-gather-4zlm2_87188b36-06f0-41e4-8cb7-bc676c308f7a/copy/0.log"
Jul 09 18:53:17.974359 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.974331    2572 generic.go:356] "Generic (PLEG): container finished" podID="87188b36-06f0-41e4-8cb7-bc676c308f7a" containerID="ab79c689f194c1e8d1827344ac42251fb351cfd87304b5fb20d7d23b2004ad9e" exitCode=143
Jul 09 18:53:17.975024 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.974483    2572 scope.go:117] "RemoveContainer" containerID="ab79c689f194c1e8d1827344ac42251fb351cfd87304b5fb20d7d23b2004ad9e"
Jul 09 18:53:17.975024 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:17.974665    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jbsj/must-gather-4zlm2"
Jul 09 18:53:18.024575 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:18.023602    2572 scope.go:117] "RemoveContainer" containerID="d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0"
Jul 09 18:53:18.065074 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:18.063022    2572 scope.go:117] "RemoveContainer" containerID="ab79c689f194c1e8d1827344ac42251fb351cfd87304b5fb20d7d23b2004ad9e"
Jul 09 18:53:18.065074 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:53:18.063426    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab79c689f194c1e8d1827344ac42251fb351cfd87304b5fb20d7d23b2004ad9e\": container with ID starting with ab79c689f194c1e8d1827344ac42251fb351cfd87304b5fb20d7d23b2004ad9e not found: ID does not exist" containerID="ab79c689f194c1e8d1827344ac42251fb351cfd87304b5fb20d7d23b2004ad9e"
Jul 09 18:53:18.065074 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:18.063465    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab79c689f194c1e8d1827344ac42251fb351cfd87304b5fb20d7d23b2004ad9e"} err="failed to get container status \"ab79c689f194c1e8d1827344ac42251fb351cfd87304b5fb20d7d23b2004ad9e\": rpc error: code = NotFound desc = could not find container \"ab79c689f194c1e8d1827344ac42251fb351cfd87304b5fb20d7d23b2004ad9e\": container with ID starting with ab79c689f194c1e8d1827344ac42251fb351cfd87304b5fb20d7d23b2004ad9e not found: ID does not exist"
Jul 09 18:53:18.065074 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:18.063491    2572 scope.go:117] "RemoveContainer" containerID="d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0"
Jul 09 18:53:18.065074 ip-10-0-130-107 kubenswrapper[2572]: E0709 18:53:18.063788    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0\": container with ID starting with d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0 not found: ID does not exist" containerID="d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0"
Jul 09 18:53:18.065074 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:18.063820    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0"} err="failed to get container status \"d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0\": rpc error: code = NotFound desc = could not find container \"d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0\": container with ID starting with d231c1aa81b13c600c137180a438128e54562afb741c93a373c7fefd63bf12f0 not found: ID does not exist"
Jul 09 18:53:19.621344 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:19.621315    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-98lvl_98cd0934-0a28-43e2-8fde-32a6dbea4697/kube-state-metrics/0.log"
Jul 09 18:53:19.643800 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:19.643769    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-98lvl_98cd0934-0a28-43e2-8fde-32a6dbea4697/kube-rbac-proxy-main/0.log"
Jul 09 18:53:19.670920 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:19.670882    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-98lvl_98cd0934-0a28-43e2-8fde-32a6dbea4697/kube-rbac-proxy-self/0.log"
Jul 09 18:53:19.763720 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:19.763690    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-s77df_d5d9c301-4af3-4b86-8c40-86876f728240/monitoring-plugin/0.log"
Jul 09 18:53:19.937997 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:19.937914    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blzln_ac47df41-c136-4bd8-a0ca-754f5809be16/node-exporter/0.log"
Jul 09 18:53:19.965772 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:19.965742    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blzln_ac47df41-c136-4bd8-a0ca-754f5809be16/kube-rbac-proxy/0.log"
Jul 09 18:53:19.990941 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:19.990912    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blzln_ac47df41-c136-4bd8-a0ca-754f5809be16/init-textfile/0.log"
Jul 09 18:53:20.124195 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.124170    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-kvvrm_8603f65d-3b7e-4e30-92fe-70e2df00c022/kube-rbac-proxy-main/0.log"
Jul 09 18:53:20.150095 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.150071    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-kvvrm_8603f65d-3b7e-4e30-92fe-70e2df00c022/kube-rbac-proxy-self/0.log"
Jul 09 18:53:20.179093 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.179062    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-kvvrm_8603f65d-3b7e-4e30-92fe-70e2df00c022/openshift-state-metrics/0.log"
Jul 09 18:53:20.230879 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.230832    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5109a0a7-ca0e-4d03-9d14-7facb16c15fb/prometheus/0.log"
Jul 09 18:53:20.253264 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.253235    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5109a0a7-ca0e-4d03-9d14-7facb16c15fb/config-reloader/0.log"
Jul 09 18:53:20.278901 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.278872    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5109a0a7-ca0e-4d03-9d14-7facb16c15fb/thanos-sidecar/0.log"
Jul 09 18:53:20.301141 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.301106    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5109a0a7-ca0e-4d03-9d14-7facb16c15fb/kube-rbac-proxy-web/0.log"
Jul 09 18:53:20.326190 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.325750    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5109a0a7-ca0e-4d03-9d14-7facb16c15fb/kube-rbac-proxy/0.log"
Jul 09 18:53:20.350291 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.350266    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5109a0a7-ca0e-4d03-9d14-7facb16c15fb/kube-rbac-proxy-thanos/0.log"
Jul 09 18:53:20.373937 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.373909    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5109a0a7-ca0e-4d03-9d14-7facb16c15fb/init-config-reloader/0.log"
Jul 09 18:53:20.409440 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.409406    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-ftgm4_acc69e44-4fc3-4a24-9a16-fa2e27ab582b/prometheus-operator/0.log"
Jul 09 18:53:20.434006 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.433978    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-ftgm4_acc69e44-4fc3-4a24-9a16-fa2e27ab582b/kube-rbac-proxy/0.log"
Jul 09 18:53:20.526022 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.525941    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f5dd465bf-nbfvs_8054ec99-6079-4530-8de8-5638d480d32c/telemeter-client/0.log"
Jul 09 18:53:20.569880 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.569855    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f5dd465bf-nbfvs_8054ec99-6079-4530-8de8-5638d480d32c/reload/0.log"
Jul 09 18:53:20.603903 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:20.603839    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f5dd465bf-nbfvs_8054ec99-6079-4530-8de8-5638d480d32c/kube-rbac-proxy/0.log"
Jul 09 18:53:22.866861 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:22.866836    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-847895f65b-qzdwj_070eb4f2-3b5e-4c4b-ade8-5a818c225987/console/0.log"
Jul 09 18:53:22.900613 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:22.900578    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-df2bx_da80f5f7-dfed-474e-96e7-c60902e0f08a/download-server/0.log"
Jul 09 18:53:23.433602 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.433572    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"]
Jul 09 18:53:23.433914 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.433902    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87188b36-06f0-41e4-8cb7-bc676c308f7a" containerName="copy"
Jul 09 18:53:23.433959 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.433941    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="87188b36-06f0-41e4-8cb7-bc676c308f7a" containerName="copy"
Jul 09 18:53:23.434000 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.433973    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87188b36-06f0-41e4-8cb7-bc676c308f7a" containerName="gather"
Jul 09 18:53:23.434000 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.433980    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="87188b36-06f0-41e4-8cb7-bc676c308f7a" containerName="gather"
Jul 09 18:53:23.434063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.434040    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="87188b36-06f0-41e4-8cb7-bc676c308f7a" containerName="copy"
Jul 09 18:53:23.434063 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.434050    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="87188b36-06f0-41e4-8cb7-bc676c308f7a" containerName="gather"
Jul 09 18:53:23.437677 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.437652    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.449541 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.449513    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"]
Jul 09 18:53:23.471386 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.471360    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c957ddbf-6c98-4635-9cad-5d0f80e35318-proc\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.471512 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.471442    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c957ddbf-6c98-4635-9cad-5d0f80e35318-sys\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.471570 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.471516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hhs5\" (UniqueName: \"kubernetes.io/projected/c957ddbf-6c98-4635-9cad-5d0f80e35318-kube-api-access-7hhs5\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.471570 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.471547    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c957ddbf-6c98-4635-9cad-5d0f80e35318-lib-modules\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.471648 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.471604    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c957ddbf-6c98-4635-9cad-5d0f80e35318-podres\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.572862 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.572823    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hhs5\" (UniqueName: \"kubernetes.io/projected/c957ddbf-6c98-4635-9cad-5d0f80e35318-kube-api-access-7hhs5\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.573028 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.572875    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c957ddbf-6c98-4635-9cad-5d0f80e35318-lib-modules\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.573028 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.572910    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c957ddbf-6c98-4635-9cad-5d0f80e35318-podres\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.573028 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.572942    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c957ddbf-6c98-4635-9cad-5d0f80e35318-proc\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.573028 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.573012    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c957ddbf-6c98-4635-9cad-5d0f80e35318-sys\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.573231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.573065    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c957ddbf-6c98-4635-9cad-5d0f80e35318-lib-modules\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.573231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.573063    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c957ddbf-6c98-4635-9cad-5d0f80e35318-podres\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.573231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.573113    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c957ddbf-6c98-4635-9cad-5d0f80e35318-sys\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.573231 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.573120    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c957ddbf-6c98-4635-9cad-5d0f80e35318-proc\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.581227 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.581198    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hhs5\" (UniqueName: \"kubernetes.io/projected/c957ddbf-6c98-4635-9cad-5d0f80e35318-kube-api-access-7hhs5\") pod \"perf-node-gather-daemonset-mq52k\" (UID: \"c957ddbf-6c98-4635-9cad-5d0f80e35318\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.749434 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.749399    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:23.922492 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:23.922457    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"]
Jul 09 18:53:23.928863 ip-10-0-130-107 kubenswrapper[2572]: W0709 18:53:23.928829    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc957ddbf_6c98_4635_9cad_5d0f80e35318.slice/crio-1411ce424272cea3a479bb82c9f68882cf1da2b53d48d035df379d171209366b WatchSource:0}: Error finding container 1411ce424272cea3a479bb82c9f68882cf1da2b53d48d035df379d171209366b: Status 404 returned error can't find the container with id 1411ce424272cea3a479bb82c9f68882cf1da2b53d48d035df379d171209366b
Jul 09 18:53:24.008700 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:24.008598    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k" event={"ID":"c957ddbf-6c98-4635-9cad-5d0f80e35318","Type":"ContainerStarted","Data":"1411ce424272cea3a479bb82c9f68882cf1da2b53d48d035df379d171209366b"}
Jul 09 18:53:24.441328 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:24.441247    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2s9k4_f2441d9d-b403-4087-842f-efc031230117/dns/0.log"
Jul 09 18:53:24.500071 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:24.500048    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2s9k4_f2441d9d-b403-4087-842f-efc031230117/kube-rbac-proxy/0.log"
Jul 09 18:53:24.735131 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:24.735105    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tgwm7_e90e2983-dbaa-4989-8e2d-9e00a858af9e/dns-node-resolver/0.log"
Jul 09 18:53:25.015240 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:25.015151    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k" event={"ID":"c957ddbf-6c98-4635-9cad-5d0f80e35318","Type":"ContainerStarted","Data":"51b48c36a15b8ea637f8a202d226351707372f6d17f411bfb9cd51d6b991f8f4"}
Jul 09 18:53:25.016219 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:25.016191    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:25.034833 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:25.034778    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k" podStartSLOduration=2.034758128 podStartE2EDuration="2.034758128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:25.033166584 +0000 UTC m=+2941.871509569" watchObservedRunningTime="2026-07-09 18:53:25.034758128 +0000 UTC m=+2941.873101113"
Jul 09 18:53:25.252732 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:25.252696    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6fbtz_c3c283f9-edab-4267-ad1b-42b1f174c884/node-ca/0.log"
Jul 09 18:53:26.522099 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:26.522076    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6m2nr_3649103d-34a2-489c-9247-c7a5e928d5d1/serve-healthcheck-canary/0.log"
Jul 09 18:53:27.113666 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:27.113635    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lzb6m_eda6d174-b6cd-4142-a416-ef8d1747ebf5/kube-rbac-proxy/0.log"
Jul 09 18:53:27.151166 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:27.151143    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lzb6m_eda6d174-b6cd-4142-a416-ef8d1747ebf5/exporter/0.log"
Jul 09 18:53:27.203051 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:27.203031    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lzb6m_eda6d174-b6cd-4142-a416-ef8d1747ebf5/extractor/0.log"
Jul 09 18:53:29.882877 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:29.882850    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9pzs7_5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982/s3-init/0.log"
Jul 09 18:53:29.907199 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:29.907171    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9pzs7_5ccc6e2a-d2e9-4a3b-90c1-a6b2aae3b982/download-hf-model/0.log"
Jul 09 18:53:29.972566 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:29.972532    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-tbb8g_7d48b26e-2420-4f74-ac50-407692b14644/seaweedfs/0.log"
Jul 09 18:53:32.036500 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:32.036475    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-mq52k"
Jul 09 18:53:35.662285 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:35.662235    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-78dlw_1e716b8d-7461-4f3f-b8e2-63a243537324/kube-multus/0.log"
Jul 09 18:53:35.874045 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:35.874018    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9r9n_8a690986-5200-4f8f-803d-1db7de256ddb/kube-multus-additional-cni-plugins/0.log"
Jul 09 18:53:35.898902 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:35.898876    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9r9n_8a690986-5200-4f8f-803d-1db7de256ddb/egress-router-binary-copy/0.log"
Jul 09 18:53:35.920537 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:35.920520    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9r9n_8a690986-5200-4f8f-803d-1db7de256ddb/cni-plugins/0.log"
Jul 09 18:53:35.943446 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:35.943426    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9r9n_8a690986-5200-4f8f-803d-1db7de256ddb/bond-cni-plugin/0.log"
Jul 09 18:53:35.966455 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:35.966434    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9r9n_8a690986-5200-4f8f-803d-1db7de256ddb/routeoverride-cni/0.log"
Jul 09 18:53:35.989925 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:35.989906    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9r9n_8a690986-5200-4f8f-803d-1db7de256ddb/whereabouts-cni-bincopy/0.log"
Jul 09 18:53:36.013222 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:36.013190    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9r9n_8a690986-5200-4f8f-803d-1db7de256ddb/whereabouts-cni/0.log"
Jul 09 18:53:36.320724 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:36.320636    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pw25w_8695fc30-cdfd-4a63-bf30-33cf14224955/network-metrics-daemon/0.log"
Jul 09 18:53:36.341021 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:36.340995    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pw25w_8695fc30-cdfd-4a63-bf30-33cf14224955/kube-rbac-proxy/0.log"
Jul 09 18:53:37.143626 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:37.143584    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-controller/0.log"
Jul 09 18:53:37.163317 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:37.163288    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/1.log"
Jul 09 18:53:37.163471 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:37.163394    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovn-acl-logging/0.log"
Jul 09 18:53:37.183907 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:37.183887    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/kube-rbac-proxy-node/0.log"
Jul 09 18:53:37.208215 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:37.208194    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 18:53:37.240357 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:37.240334    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/northd/0.log"
Jul 09 18:53:37.265070 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:37.265034    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/nbdb/0.log"
Jul 09 18:53:37.293962 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:37.293940    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/sbdb/0.log"
Jul 09 18:53:37.347774 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:37.347753    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmxsn_a6c1dc49-8503-40c7-b286-0d1b9706aec6/ovnkube-controller/0.log"
Jul 09 18:53:38.993005 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:38.992977    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-c9jnd_d50047de-b0e9-4a3c-99ad-b4566e13172c/network-check-target-container/0.log"
Jul 09 18:53:40.036196 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:40.036168    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-bnnxp_a0281d37-6c54-407c-bc65-61d01eeb63b9/iptables-alerter/0.log"
Jul 09 18:53:40.744280 ip-10-0-130-107 kubenswrapper[2572]: I0709 18:53:40.744252    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-hg7xk_3ea667e0-903b-43e7-b192-0ba95ac6c1f9/tuned/0.log"